# The Rise of Modern Data Management

Chad Sanderson, Gable.ai | MLOps Podcast | Episode 226 | 57:53
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=h5agcL6t4uM
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/the-rise-of-modern-data-management
Published: 2024-04-24
Tags: data-engineering, data-quality

## TL;DR
- Companies are moving away from building data infrastructure before they know how it will create business value.
- Data changes need software-style change management because failures can affect machine learning models, invoice reconciliation, and other operational systems.
- Data contracts can connect data producers and consumers through ownership, expectations, lineage, and feedback inside existing development workflows.

## Summary
Chad Sanderson argues that data infrastructure is changing because companies are demanding a clearer link between data and business value. Analytics pipelines built around Snowflake, data lakes, Fivetran, dbt, and Looker often leave producers without responsibility for the data they create. That model becomes dangerous when the data feeds machine learning or operational processes. A schema or content change can damage a model, break reconciliation, or force data engineers into weeks of investigation and backfills. Sanderson compares the old centralized model to a library and the emerging model to a bookstore, where producers make their data available directly. Gable.ai applies software development ideas to data through source registration, ownership, lineage, change detection, expectations, and CI/CD checks. Sanderson says producers need feedback in the tools they already use, especially GitHub and GitLab. He expects data management to become part of engineering workflows as companies rely more heavily on data to make money or reduce costs.

## Key ideas
### Data teams are being asked to prove value before building large platforms
[04:59](https://www.youtube.com/watch?v=h5agcL6t4uM&t=299s)
Chad Sanderson says the early-2020s data bubble encouraged startups to spend heavily on Snowflake, Databricks, and many other tools without knowing what the data would deliver. Companies are now asking data teams to explain how data will be used and how it will produce return on investment before the infrastructure exists. He connects this shift to older data warehouse projects, where storage, compute, architecture, and staffing were expensive enough to force careful planning. Analytics can inform product decisions, but Sanderson distinguishes it from operational uses such as invoice reconciliation and machine learning, where data failures can directly affect the business.

### Analytics-first pipelines leave producers disconnected from data quality
[08:27](https://www.youtube.com/watch?v=h5agcL6t4uM&t=507s)
Sanderson describes a common early-stage setup with Snowflake, a data lake, Fivetran, dbt, and Looker. Product teams often build this infrastructure for analytics rather than operational systems. The problem begins when extraction tools pull data from engineers' services even though those engineers do not treat the data as a product or API. A change that merely fixes one service can break a downstream marketing workflow, model, or other consumer. In analytics, the team may repair a dashboard after the fact. In operational systems, the same delay can cause financial consequences.

### Data ownership has to move toward the people who produce the data
[11:10](https://www.youtube.com/watch?v=h5agcL6t4uM&t=670s)
Sanderson compares centralized data management with a library. Librarians control what enters the collection, organize it, and track what leaves. That model becomes difficult when a company has hundreds of thousands or millions of data sets. The alternative is federated management, which he compares with a bookstore or Amazon. Engineers, salespeople, and other producers make data available, while consumers can understand and use it. Sanderson also explains why the bookstore metaphor has limits: a producer may create data to run an application, without intending other teams to consume it. The resulting problem is usually lack of awareness rather than lack of care.

### Data creates dependency chains that cannot be decoupled like application services
[16:46](https://www.youtube.com/watch?v=h5agcL6t4uM&t=1006s)
Sanderson says a metric such as profit depends on revenue and cost, while revenue depends on several underlying sources and line items. These dependency trees make data resemble a supply chain. Logs or events are transformed through raw and normalized layers, domains, and metrics. A break at any point can affect every downstream user. Software teams already manage code changes with unit tests, integration tests, pull requests, diffs, deployment controls, and feature flags. Data changes happen just as often, through altered database fields or added and removed events, yet data teams often discover problems only after a model or dashboard fails.

### Data engineers become bottlenecks because failures are investigated after they happen
[21:19](https://www.youtube.com/watch?v=h5agcL6t4uM&t=1279s)
When a workflow breaks, data scientists and analysts often escalate the issue to data engineering. The data engineer traces lineage, searches pull requests, contacts code owners, argues for a rollback or fix, and then performs a backfill. The producer may not see the issue as urgent, so the fix can sit on a backlog for weeks or months. Data consumers also spend time validating whether data has the expected meaning and whether changes will damage their work. Sanderson contrasts this with software engineers consuming a documented API. Data is also an interface, he says, but consumers often receive none of the contract, ownership, documentation, or guarantees that an API provides.

### Gable.ai detects data sources and changes before they reach consumers
[26:29](https://www.youtube.com/watch?v=h5agcL6t4uM&t=1589s)
Sanderson describes Gable.ai as a platform that builds a consistent abstraction across many data sources. It identifies sources in code, including database structures and event code, and can scan file systems and third-party systems. It extracts schemas, semantics, and data contents into a standard format. By connecting with GitHub or GitLab, Gable.ai can identify code ownership, track asset changes, and build relationships between assets. Humans then add expectations such as allowed values, valid ranges, and event-volume SLAs. The platform can detect structural changes in CI/CD and warn about content changes before pipelines run and affect a model.

### Data contracts are also becoming catalogs and collaboration systems
[39:55](https://www.youtube.com/watch?v=h5agcL6t4uM&t=2395s)
Sanderson originally focused on using data contracts to bring data into the DevOps process through automated scanning, ownership information, constraints, and CI/CD checks. Conversations with companies showed that they wanted more. They wanted source data cataloged in one place, organized by domain, and monitored for live changes. They wanted to know when someone changed an Excel file, so platform teams could protect consumers before the change reached an analytical database. Sanderson says the aim is to move as far upstream as possible, including code and systems such as Excel, Google Sheets, Salesforce, and SAP. Some human-entered values cannot be caught in CI/CD, so detection must also happen when the data becomes accessible.

### Producers respond when data management fits their existing engineering workflow
[47:13](https://www.youtube.com/watch?v=h5agcL6t4uM&t=2833s)
Sanderson says he expected data teams to care most about Gable.ai, but software engineers have also responded strongly. Producers often understand that changes cause problems, yet lack a practical mechanism for helping data consumers. They may not know Snowflake, Databricks, data lakes, ETL systems, or dbt, and they may not know how to investigate a failure. Gable.ai frames the work as another DevOps step. A producer can see which colleagues depend on records, what those colleagues use them for, and what needs to happen before a pull request moves forward. Sanderson expects data management to become part of engineering workflows in the same way security has become part of software development.

## Notable quotes
- Chad Sanderson: "Data is an API too." (24:24)
- Chad Sanderson: "Data is changing all the time yet there is no system for the teams actually leveraging that data to give any sort of human in the loop review or feedback." (20:23)
- Chad Sanderson: "The problem I feel is not that these product engineers don't care, it's that they don't know." (15:35)
- Chad Sanderson: "It's applying the AI magic to bring the people together and have closer connectivity when it comes to managing and creating data in the first place." (39:33)
- Chad Sanderson: "If you don't do data management from the beginning, data quality issues will happen and you just have to deal with it." (52:07)

## Tools & references mentioned
- Gable.ai
- Gable
- Convoy
- Microsoft
- Sephora
- Subway
- Oracle
- Snowflake
- Databricks
- Fivetran
- dbt
- Looker
- Kafka
- GitHub
- GitLab
- LaunchDarkly
- Amplitude
- Mixpanel
- AWS Trainium
- AWS Inferentia
- PyTorch
- TensorFlow
- Salesforce
- SAP
- Excel
- Google Sheets

## Who should watch
- You own data pipelines that feed machine learning models or operational processes and need to catch producer-side changes before they cause damage.
- Your data engineering team spends its time tracing lineage, contacting service owners, fixing broken workflows, and running backfills.
- You are introducing data contracts and want them to fit into GitHub, GitLab, pull requests, and CI/CD rather than creating a separate process for producers.

## Related talks

- [Data Engineering for ML](https://mlopstalks.com/talks/data-engineering-for-ml) (Chad Sanderson, Convoy, 57:54)
- [Data Contracts: The Missing Piece of the Data Puzzle](https://mlopstalks.com/talks/data-contracts-the-missing-piece-of-the-data-puzzle) (Mark Freeman, Humu, 13:40)
- [The Only Constant is (Data) Change](https://mlopstalks.com/talks/the-only-constant-is-data-change) (Benjamin Rogojan, Seattle Data Guy & Chad Sanderson, Gable & Christophe Blefari, NAO & Maggie Hays, Acryl Data, 40:50)
- [Driving ML Data Quality with Data Contracts](https://mlopstalks.com/talks/driving-ml-data-quality-with-data-contracts) (Andrew Jones, GoCardless, 34:30)
- [Data Engineering + ML + Software Engineering](https://mlopstalks.com/talks/data-engineering-ml-software-engineering) (Satish Chandra Gupta, Slang Labs, 57:05)