# Data Contracts: The Missing Piece of the Data Puzzle Mark Freeman, Humu | MLOps Community | 13:40 Source: https://www.youtube.com/watch?v=7rfWgmDRCWU Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks. Page: https://mlopstalks.com/talks/data-contracts-the-missing-piece-of-the-data-puzzle Published: 2024-10-09 Tags: data-engineering, data-quality, observability ## TL;DR - Data contracts prevent known data quality problems before a change reaches a data asset. - Data observability detects problems after they occur across the wider data system. - Teams should use observability to find important problem areas, then apply contracts to critical workflows. ## Summary Mark Freeman explains why data contracts and data observability address different parts of data quality. Observability gives teams broad visibility across pipelines, infrastructure, dependencies, and data changes. It can reveal problems as they occur, such as predictions moving outside an expected range. Data contracts work earlier in the process. They record expectations about a data asset, including its schema and business rules, then check proposed changes through code review and CI/CD. A contract can block a high-impact change or allow a lower-risk change while prompting a discussion. Freeman compares observability to a flashlight that scans the data system and contracts to a laser pointer aimed at selected workflows. He recommends starting with broad monitoring to find trouble, then using contracts where the consequences of change matter most. He also describes an emerging implementation pattern based on a contract specification, JSON Schema, version control, CI/CD, and a data catalog. ## Key ideas ### Data quality problems often begin with a communication gap between producers and consumers [01:14](https://www.youtube.com/watch?v=7rfWgmDRCWU&t=74s) Mark Freeman describes moving from data science into data engineering after finding that startup data was difficult to use and lacked supporting infrastructure. Working between software engineering and data science exposed a large gap in how the groups thought about data. He joined Gable to work on this problem with Chad Sørenston, focusing on how teams could agree on expectations for data before changes caused downstream problems. The talk is based on their upcoming O'Reilly book about data contracts. ### Data contracts and observability can be placed at many points in a data system [02:52](https://www.youtube.com/watch?v=7rfWgmDRCWU&t=172s) Freeman sketches a data stack that includes a transactional database, an analytical database, and systems used for analytics or machine learning. Each connection between a source and a target can be a place for a data contract or observability. He says teams often ask where each approach belongs and whether they should apply both everywhere. His practical distinction is that contracts tend to work upstream, while observability tends to watch downstream effects. ### Observability finds data problems across the system after changes happen [03:51](https://www.youtube.com/watch?v=7rfWgmDRCWU&t=231s) Freeman defines data observability as broad visibility into data, pipelines, infrastructure, and dependencies, with the aim of identifying and addressing data issues within acceptable service levels. It can alert a team when predictions move outside an expected range or when data changes appear. The important timing difference is that observability reports what has happened or is happening. It does not normally prevent a change before the data is written. ### A data contract records expectations and checks them before a change is accepted [04:51](https://www.youtube.com/watch?v=7rfWgmDRCWU&t=291s) A data contract extends software engineering collaboration practices to data teams. A contract can describe the expected schema and business rules for an analytical table, transactional database, event stream, or external source such as Salesforce. Freeman describes a contract specification, often written in YAML, being checked against a schema registry or data catalog. Detection can use change data capture, stream processing, lineage, static analysis, or live monitoring. The result feeds into CI/CD, version control, and alerts. ### Contracts preserve agreements when people and responsibilities change [06:54](https://www.youtube.com/watch?v=7rfWgmDRCWU&t=414s) Freeman gives an example in which a data scientist needs a particular format for a machine learning model and asks an upstream producer to agree to it. The producer may leave the company, and the replacement may not know about the original agreement. If the contract is stored as code in version control, future changes still pass through the same checks. A failed check can block a high-impact change or allow a lower-impact change while creating a prompt for people to discuss it. ### Contracts prevent targeted problems while observability reveals wider patterns [08:45](https://www.youtube.com/watch?v=7rfWgmDRCWU&t=525s) Freeman separates the two approaches by timing, scope, and workflow. Data contracts prevent specific quality problems and fit inside CI/CD. Data observability complements CI/CD by showing quality trends across the full data system. Contracts are informed by business logic and provide targeted visibility. Observability shows how the system is behaving more broadly. The simplest timing distinction is that contracts alert before a change, while observability alerts after a change. ### Teams should use observability to find where contracts deserve attention [09:47](https://www.youtube.com/watch?v=7rfWgmDRCWU&t=587s) Freeman compares observability to a flashlight and contracts to a laser pointer. Applying contracts to every data asset would require too much effort because systems contain many assets and rules. Observability can first scan the system and show what is working and what is failing. Once a team understands which parts matter most, it can apply contracts to a critical workflow and get the relevant producers and consumers aligned. ### The implementation pattern is still emerging and requires several components [11:40](https://www.youtube.com/watch?v=7rfWgmDRCWU&t=700s) In the question period, Freeman says there is no single all-encompassing open-source tool for data contracts. Teams may combine a contract specification with JSON Schema, GitHub Actions or another CI/CD system, their existing data assets, and a data catalog. He describes JSON Schema as flexible and supported by plugins for different languages. He also says standardization is relatively new, so teams currently need to assemble the pieces around an architectural pattern. ## Notable quotes - "My argument is, hint, yes, yes we do." (01:14) - "What if you can be notified before data is even written? And that's what data contracts are." (04:34) - "Data contracts are preventing specific data quality issues while data observability is highlighting data quality trends." (08:45) - "You have the flashlight, you can shine over the entire different data system and understand what is wrong, what is working, and then once you have an idea of, okay, these are the most important pieces, take the laser pointer with data contract." (10:19) - "I would consider data contracts more data contracts architecture pattern." (12:18) ## Tools & references mentioned - Mark Freeman - Chad Sørenston - Gable - O'Reilly - Gartner - Salesforce - GitHub - JSON Schema - GitHub Actions - Snowflake - Databricks - DataHub ## Who should watch - You are deciding whether data contracts, data observability, or both belong in your data quality work. - Your team has recurring disagreements between data producers and consumers about schemas, business rules, or downstream use. - You need a starting architecture and want to understand how contracts could fit with CI/CD, JSON Schema, and a data catalog. ## Editor's note From the pack [Data quality for machine learning](https://mlopstalks.com/packs/data-quality-for-machine-learning): The opening talk traces quality failures backwards through transformations, while the Convoy panel shows why a successful data job can still produce harmful inputs. ZenML records each pipeline run's steps, inputs, outputs and code version, giving an investigation a path back to the data and code that produced an artifact. Those records complement the source agreements and quality checks described here. Written by the MLOps Talks editors (the ZenML team), not by the speaker. ## Related talks - [Driving ML Data Quality with Data Contracts](https://mlopstalks.com/talks/driving-ml-data-quality-with-data-contracts) (Andrew Jones, GoCardless, 34:30) - [The Rise of Modern Data Management](https://mlopstalks.com/talks/the-rise-of-modern-data-management) (Chad Sanderson, Gable.ai, 57:53) - [Trustworthy Data for Machine Learning](https://mlopstalks.com/talks/trustworthy-data-for-machine-learning) (Chad Sanderson, Convoy, 51:04) - [Data Quality = Quality AI](https://mlopstalks.com/talks/data-quality-quality-ai) (Samuel Partee, Redis & Chad Sanderson, Gable & Joe Reis, Ternary Data & Maria Zhang, Proactive AI Lab Inc & Pushkar Garg, Clari Inc, 27:15) - [The intersection between DataOps and privacy](https://mlopstalks.com/talks/the-intersection-between-dataops-and-privacy) (Lars Albertsson, Scling, 33:26)