# Data Quality = Quality AI

Samuel Partee, Redis & Chad Sanderson, Gable & Joe Reis, Ternary Data & Maria Zhang, Proactive AI Lab Inc & Pushkar Garg, Clari Inc | AIQCON 2024 | 27:15
Hosted by Sam Partee

Source: https://www.youtube.com/watch?v=r2fxb-OnBRM
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/data-quality-quality-ai
Published: 2024-08-16
Tags: data-engineering, data-pipelines, data-quality, platform-teams

## TL;DR
- Data quality needs measurable checks for completeness, accuracy, validity, and timeliness, followed by ongoing monitoring.
- A schema change and machine-generated auction bids caused serious failures in Convoy's pricing model because the data's meaning and source behavior were not understood.
- Data contracts, upstream checks, shared platform tools, and closer cooperation between data producers and consumers can reduce failures before they reach AI systems.

## Summary
The panel defines data quality as a set of measurable properties that need continuous monitoring. Maria Zhang names completeness, accuracy, validity, and timeliness. Pushkar Garg separates the data side, where ingestion and transformation happen, from the machine learning side, where features become models and predictions. Chad Sanderson describes two Convoy incidents: a dropped feature broke a pipeline, while machine-generated bids contaminated training data and contributed to a major business loss. The panel says companies often rush toward generative AI while their structured data practices remain weak. Speakers recommend understanding data at a granular level, documenting metadata and institutional knowledge, checking data close to its source, and using platform-level tools in pipelines. Sanderson proposes data contracts that define schemas, quality rules, expected volumes, and service levels between producers and consumers. Joe Reis connects this work to lean supply-chain practices, while Maria argues that people must take ownership of how data affects downstream systems.

## Key ideas
### Data quality needs explicit metrics and monitoring
[00:50](https://www.youtube.com/watch?v=r2fxb-OnBRM&t=50s)
Maria Zhang says data quality changes with the situation, so teams need to define what it means for their use case. She names data completeness, accuracy, validity, and timeliness as measures that can support a monitoring structure. Pushkar Garg adds that teams should bring structured and unstructured data into a shared lake or lakehouse before monitoring it. He separates ingestion and transformation from the later work of turning data into features, models, and predictions. The panel treats monitoring as an ongoing operating practice rather than a one-time check. Without it, teams can discover problems only after a model, dashboard, or AI program has already failed.

### Ignoring data quality makes failures hard to explain and fund
[02:15](https://www.youtube.com/watch?v=r2fxb-OnBRM&t=135s)
Chad Sanderson says failures become difficult to diagnose at scale. Models fail, dashboards show wrong information, and AI programs stall, while executives ask for the return on investment from fixing data quality. Joe Reis describes this as the normal state in many companies. The panel also asks the audience whether their data is pristine, and very few people raise their hands. Their point is practical: if quality work starts only after an incident, the team must investigate without the context, ownership, or checks that could have prevented it. The cost then includes delayed diagnosis and a harder case for investment.

### Small changes in data behavior can damage an important model
[05:22](https://www.youtube.com/watch?v=r2fxb-OnBRM&t=322s)
Chad Sanderson recounts two incidents from Convoy's pricing model. A junior engineer removed a column called "is dropped" because the application no longer used it, without knowing that the model depended on it. The change broke the data pipeline, and the cause was unclear for several days. A second incident came from Auto bid, which created machine-generated bids in a marketplace where the model had been trained on human bidding behavior. After a period of time, thousands of those bids entered the training data. The panel uses the example to show that data quality includes changes in how data is produced, not only missing values or invalid schemas.

### Enterprise readiness requires standards for accuracy, service levels, and safety
[08:38](https://www.youtube.com/watch?v=r2fxb-OnBRM&t=518s)
Maria Zhang says generative AI creates an open question about what enterprise readiness means. Teams must decide how to define compliance, service-level agreements, hallucination, misalignment, and acceptable error. She questions whether first-token arrival satisfies an SLA and whether a high error tolerance is acceptable when agents cannot reach perfect accuracy. Joe Reis describes the current enterprise data environment as poor enough that many companies struggle with basic business intelligence. The panel argues that putting a language model over corporate data without confidence in the underlying data can produce misleading results. Enterprise AI therefore needs agreed standards that cover both system behavior and data quality.

### Teams should fix basic structured-data problems before expanding to unstructured AI
[12:08](https://www.youtube.com/watch?v=r2fxb-OnBRM&t=728s)
Chad Sanderson says relational data quality is easier than unstructured-data quality, yet many companies still struggle with basic schema changes. He also points to semantic changes that leave the schema intact, such as changing a timestamp's meaning or representation. Pushkar Garg says predictive and generative systems share some quality concerns, but unstructured inputs such as emails and call transcripts require additional work to identify tone or actions. The panel warns against treating every problem as an LLM problem. Joe Reis notes that intent classification could be handled with conventional methods such as XGBoost when the task is simply assigning defined categories.

### Understanding data at column level makes quality work manageable
[16:30](https://www.youtube.com/watch?v=r2fxb-OnBRM&t=990s)
Maria Zhang recommends examining data in granular terms instead of placing everything into one large category such as structured or unstructured. Teams should distinguish dynamic data, slow-changing factual data, and human-labeled data with subjectivity. Each type needs different checks. Dynamic data may require latency and completeness measures. Slow-changing factual data may need change detection, so an unexpected large change triggers investigation. Chad Sanderson connects this work to metadata: teams need to know where data comes from, what it means, who uses it, and how it changes. Without documentation, data scientists depend on undocumented institutional knowledge held by the engineers who created the data.

### Platform teams can make quality checks part of normal pipeline work
[20:03](https://www.youtube.com/watch?v=r2fxb-OnBRM&t=1203s)
Pushkar Garg recommends implementing quality controls at the platform level and exposing them through shared frameworks. Data pipelines can include operators in Airflow or another orchestration tool, along with tools such as Great Expectations. Data scientists then use these checks while building pipelines instead of creating every control from scratch. Garg frames this as a platform team's responsibility to make the safe path easier. The approach also reduces dependence on each individual data scientist knowing infrastructure tools. The panel's emphasis is on putting checks into the systems people already use, close to the point where data is ingested and transformed.

### Data contracts connect producers and consumers across a data supply chain
[21:36](https://www.youtube.com/watch?v=r2fxb-OnBRM&t=1296s)
Chad Sanderson compares a data pipeline with a supply chain. Raw data is transformed through several steps before becoming a data product, including an AI system. Each producer and consumer needs visibility into how a change will affect downstream users. He recommends data contracts that define expected schemas, contents, quality rules, service levels, and event volumes. The check should happen as close to the producer as possible, so a failure can be communicated to downstream contracts. Joe Reis links this approach to lean ideas such as continuous flow, fewer bottlenecks, and catching errors before a large batch needs rework. Maria Zhang adds that people must understand the business effect of the data they create.

## Notable quotes
- Maria Zhang: "Data quality for me, it needs to be available on certain metrics and then those metrics need to be monitored." (01:10)
- Chad Sanderson: "We had two really major data quality incidents within a period of about three months." (06:12)
- Chad Sanderson: "I think there are so many companies that are trying to start thinking about investing heavily in AI and AI quality, but their ability to do data quality on relational data sets is horrible." (13:29)
- Chad Sanderson: "The data contract is basically starting to apply an API to your data." (22:28)
- Maria Zhang: "Take a closer look and really break it down." (17:07)

## Tools & references mentioned
- Convoy
- XGBoost
- Airflow
- Great Expectations
- Snowflake
- The Goal
- Deming

## Who should watch
- You are building an AI or machine learning platform and need quality checks that data scientists can use inside ordinary pipelines.
- Your company wants to put generative AI over internal data, but basic schemas, metadata, ownership, or business intelligence are still unreliable.
- Your team has experienced incidents caused by upstream changes and needs a practical model for contracts between data producers and consumers.

## Editor's note

Pushkar Garg recommends putting data-quality controls into shared pipeline frameworks so teams do not create every check from scratch. ZenML records each pipeline run's steps, inputs, outputs, and code version, giving teams a record of how a model or artifact was produced when a schema or source-behavior change causes a failure.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [Data Quality Management Techniques - The Complete Guide](https://mlopstalks.com/talks/data-quality-management-techniques-the-complete-guide) (, 28:07)
- [Driving ML Data Quality with Data Contracts](https://mlopstalks.com/talks/driving-ml-data-quality-with-data-contracts) (Andrew Jones, GoCardless, 34:30)
- [Data Selection for Data-Centric AI: Data Quality Over Quantity](https://mlopstalks.com/talks/data-selection-for-data-centric-ai-data-quality-over-quantity) (Cody Coleman, 1:11:01)
- [Managing Data for Effective GenAI Application](https://mlopstalks.com/talks/managing-data-for-effective-genai-application) (Anu Arora & Anass Bensrhir, QuantumBlack AI by McKinsey, 51:01)
- [Eliminating Garbage In/Garbage Out for Analytics and ML](https://mlopstalks.com/talks/eliminating-garbage-in-garbage-out-for-analytics-and-ml) (Roy Hasson & Santona Tuli, Upsolver, 50:38)