Production data science needs trusted datasets because model outputs can affect customers, revenue, and legal risk.
2
Feature engineering platforms need metadata, lineage, reproducibility, checkpointing, and data-quality checks to support debugging and accountability.
3
The right feature engineering architecture depends on data shape, team size, model count, and how closely features are tied to modeling.
Summary
Dr. Venkata Pingali explains why feature engineering needs the same discipline applied to software systems. His concern began with marketing projects where teams optimized for speed without checking whether the data and resulting claims were correct. In production, a model can affect inventory, pricing, customer decisions, and revenue, so teams need to know what data was used, when it was computed, and how it changed. Scribble Data built these capabilities in response to customer needs, including metadata, lineage, reproducibility, checkpointing, and quality monitoring. Pingali argues that feature engineering is not one linear step before model training. It touches resource planning, validation, workflow management, and parts of serving. The architecture also changes with data shape and organizational scale. Once an organization has around eight to ten models or use cases, he expects a shared feature platform to become necessary. He also describes the systems knowledge an ML engineer needs to work with data scientists and data engineers.
Production data science needs trust and accountability
Pingali traces his focus on production systems to data projects where speed mattered more than correctness. Teams shared model-based claims with stakeholders without asking whether the claims were accurate. That becomes unacceptable when a CFO uses data to reject customers or when a system changes real-world decisions. He says data science is moving from a laboratory activity into a regular organizational function that finance, operations, and legal teams must be able to depend on. Explainability therefore reaches beyond the model. It also requires answers about the underlying data, the transformations that produced it, and the people responsible for each step.
Lineage and reproducibility help teams debug their own systems
Pingali gives an inventory forecasting example where unusual model behavior was eventually traced to Java application logic that handled products differently by geography. The team first inspected modeling code and a Jupyter notebook, then followed the dataset back through the data pipeline. Scribble Data's metadata and lineage records showed which sources produced the dataset, allowing the team to trace the problem to the Java source and fix it. He says this kind of debugging need will drive reproducibility and explainability even without regulatory pressure. Teams cannot manage risk if they do not know what their data system did, when it did it, or why.
Checkpointing turns data-service promises into operational capabilities
Scribble Data added reproducibility and checkpointing because customers wanted to treat the platform as a data service. If a bug enters a transformation, the team must identify when it began and how far its effects propagated. That requires knowing the state of a pipeline at a particular checkpoint and being able to backfill from the right point. Pingali connects these capabilities with metadata standardization and data-quality monitoring. The platform's promise of confidence in data required more than generating a final dataset. It required records of what happened throughout preparation, including the raw inputs and intermediate states.
Pingali says metadata, lineage, and data quality are connected. A team cannot claim confidence in a prepared dataset without confidence in the raw data from which it was built. Scribble Data initially encountered monitoring as an unexpected use case because customers wanted to check existing datasets, not only create new ones. He argues that data quality is needed for trust, and trust supports explainability and auditability. The model code is becoming more predictable through standard libraries, while the training data remains a major source of unpredictable behavior. For that reason, he expects trust in data to receive more attention as models affect business decisions.
The surrounding infrastructure often takes more work than model building
Pingali discusses a Google paper from 2015 that showed machine learning systems require substantial infrastructure around the model. Scribble Data began with a simple view of feature engineering as a linear process that feeds data into a model. Deployments showed a wider set of needs, including data monitoring, quality checks, resource management, efficiency work, and parts of serving infrastructure. He describes four goals for the overall system: increase the speed of adding use cases, give stakeholders confidence, support continuous change and debugging, and scale across more customers and products. The exact distribution of effort differs by company size and system complexity.
Feature engineering has several valid design points
Pingali rejects the assumption that every company should copy the architecture of Uber or Google. He says there are at least six or seven possible design points even within feature engineering. A system for data with ten columns and ten billion rows can look different from one for five million rows and two hundred columns. The relationship between feature preparation and deep learning also changes the architecture. As organizations grow, feature engineering often starts as one integrated activity and then develops clearer boundaries between data collection, preparation, and modeling. Pingali says a standalone shared platform is generally needed after an organization reaches about eight or ten models or use cases, because inconsistent feature definitions create confusion and slow development.
Feature pipelines need careful planning because errors propagate
Feature preparation pipelines can form a chain in which one pipeline feeds another. If a leaf-level variable is defined incorrectly, the error can affect data science work over the following months. Pingali says productionization exposes hidden assumptions and bottlenecks that are easy to miss during modeling. He recommends thinking through these dependencies before the modeling phase is complete. The platform must give teams control over definitions, sources, quality checks, and downstream effects. He also notes that teams with roughly twenty or thirty people may struggle to assemble all these pieces, since they have limited resources and need to decide which processes and interfaces to standardize.
In response to a question about overlapping feature sets, Pingali describes a multi-step feature chain. Users can take a prepared feature set when it fits their use case, or move one level deeper when they need additional information. In Scribble Data deployments, partially computed information is available at several levels before the final output reaches the user. A use case may start at the fourth level, return to the third, or go back to raw data when another attribute must be extracted. He does not favor generating every possible feature combination. Data scientists usually know which features they want, and the computational cost can be too high for large feature matrices.
Orchestration matters less than the structure of the development process
Scribble Data integrates with Airflow and uses Prefect for cross-process workflows, while maintaining an internal workflow system for operations at data-frame granularity inside a process. Pingali says the orchestration tool itself is less important than how the team structures development, modules, and operating practices. He has mixed feelings about Kubeflow and Kubernetes for customers that cannot dedicate development and operations teams to running them. In those cases, Scribble Data has used simpler deployments. His answer places orchestration inside a broader system-design problem rather than treating it as the central solution.
"The problem is not so much on the computer building blocks. Even if it is PySpark, you need to know what is going into Python, what is coming out of PySpark, how did you do it, and are you doing a quality check before it goes in and after it comes out."Dr. Venkata Pingali27:24
Who should watch
You are building feature pipelines that need to be reproducible, auditable, or explainable when a model behaves unexpectedly.
Your team is deciding whether feature engineering should remain inside individual projects or become a shared platform.
You need to choose an architecture for a mid-sized data team and want guidance grounded in data shape, workflow complexity, and operational constraints.
