Cloud platforms reduced the cost of storage and compute, but they created governance problems because more teams can produce and distribute data independently.
2
Data models accumulate more easily than software features because teams rarely measure their use or have a clear process for deprecating them.
3
Data teams will gain clearer accountability when pipelines and models are connected to products, users, and measurable business value.
Summary
The panel traces how data engineering changed from centrally managed databases and warehouses to cloud systems where many teams can create, transform, and share data. The panelists argue that storage and compute became easier to access, while governance, ownership, documentation, and quality became harder. They discuss the tension between letting analysts move quickly and preventing thousands of one-off models and pipelines from becoming permanent dependencies. Data models also differ from software features because they often begin as answers to questions, without a clear way to measure value or decide when they should be removed. The panel recommends treating data development as an ongoing product practice, with environments, ownership, model pruning, and deprecation strategies. Chad Sanderson connects this to data products with a user-facing output and a pipeline behind it. The panel is candid that data teams still struggle to connect their work to business value.
Data engineering began with centralized systems and slow access
Maggie Hays recalls working with SAS and Teradata at Bank of America, where a central database team managed the systems and the data appeared ready for analysis without much discussion of how it was created. During the first days of each month, batch backfills made the database unavailable for queries. Chad Sanderson describes Excel, SSIS, relational databases, and FTP data dumps, with quality work focused on preventing people from changing schemas. Benjamin Rogojan began with SQL Server in a hospital and initially understood the warehouse as an ordinary database.
Cloud removed the cost barrier and spread data production across teams
Chad Sanderson says the cloud made storage and compute accessible to far more companies than the old Teradata-style world. Chad argues that this solved one problem while creating another: teams can now push data into cloud storage without others knowing where it came from or how good it is. Maggie Hays adds that companies can connect tools and create new data resources much faster than their governance and compliance processes can handle. The result is a federated environment where the old central data steward cannot manage everything alone.
Christophe Blefari argues that many data teams remain in an existential crisis because they have not found a clear purpose inside the company. He says teams often agree to every request, work under time pressure, and create poorly designed models or missing contracts while trying to prove their value. The panel agrees that technology changed faster than the people and organizational problems. Data teams can now produce results more quickly, but speed also makes it easier to create assets that lack ownership, structure, or a clear reason to exist.
Governance has to work in a federated data environment
Chad Sanderson describes four layers in a data organization: storage and compute, data storage and querying, data movement, and management systems for quality, governance, and compliance. Data applications sit above these layers and use the managed data for business tasks. He says cloud companies such as dbt, Fivetran, and Snowflake succeeded by rethinking storage, transformation, and orchestration for a federated system. Data management needs the same kind of redesign rather than simply copying practices from the on-premises era.
Christophe Blefari says dbt makes it easy for analysts to create governed data assets, but that ease can produce a large amount of entropy. A team of ten analysts creating several models each week can have a serious maintenance problem by the end of the year. He argues that someone eventually has to stop new work and refactor the system. Maggie Hays adds that data models lack the usage signals available for application features, so unused models can remain live for years while teams hesitate to remove them because a stakeholder may still need them.
Data teams need explicit pruning and deprecation practices
The panel treats model removal as a form of technical debt that does not fit neatly into the usual software engineering process. Maggie Hays describes cycling ownership of data modeling at Braintree. The practice continually refined core models and exposed team members to different parts of the business, reducing dependence on one person. It also required time and could slow stakeholder requests, so the team had to explain why the investment mattered. The panel suggests that development environments should be paired with a clear strategy for deprecating models.
Data products can connect pipelines to business value
Chad Sanderson says data will be treated more like a product when teams can draw a clear line from a model to money in the bank. He describes a data product as having a front end, such as a dashboard or machine learning prediction, and a back end made of pipelines and models. This framing gives data teams a way to identify which pipelines create value for customers. The panel expects software engineering and data work to converge partly because both will become more experimental, while data teams gain better accountability for the products they support.
"I think that data products will effectively have that same sort of relationship where you've got a front end and the front end might be a dashboard, it might be the prediction of a machine learning model, and the back end is your pipeline."Chad Sanderson39:15
Who should watch
You manage a data platform where analysts and business teams can create models faster than the platform team can review them.
Your organization has many pipelines with unclear ownership, weak documentation, or no agreed process for retiring unused assets.
You want to connect data models and pipelines to customer-facing products or measurable business outcomes.
