# The Future of Data: Composability & the Modular Data Stack

Stephen Bailey, Whatnot | MLOps Community | 13:11

Source: https://www.youtube.com/watch?v=n0sVot6s3-E
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/the-future-of-data-composability-the-modular-data-stack
Published: 2024-10-09
Tags: data-engineering, monitoring, orchestration, platform-teams

## TL;DR
- Whatnot found that reducing data complexity requires consistent patterns for deploying, monitoring, and managing every kind of data service.
- A data service should expose a simple interface, while its implementation details remain behind contracts, guardrails, and a shared control plane.
- Data mesh offers useful principles, but Stephen Bailey argues that teams also need practical deployment and management patterns for data services.

## Summary
Stephen Bailey describes Whatnot's effort to scale its data platform as the company added analytics, machine learning, real-time systems, and application use cases. A simple medallion architecture gradually became a complex collection of warehouses, streaming systems, experimentation tools, operational integrations, and real-time databases. Bailey uses data mesh as a useful reference, especially its ideas about data products, self-service platforms, domain ownership, and system-level governance. He also explains why full decentralization can be too heavy for a small or rapidly changing organization. Whatnot's response is to create repeatable patterns for data services. These patterns include clear service names, self-service interfaces with guardrails, end-to-end ownership of interfaces, a central control plane, shared infrastructure practices, monitoring, and service-level objectives. The event bus illustrates the approach: producers and consumers interact through simple interfaces and Protobuf contracts, while Dagster, DataDog, and Terraform provide common management tools. Bailey argues that data infrastructure is still in a pre-Docker stage, with too little modularity around data movement and pipelines.

## Key ideas
### Whatnot's growing data stack created operational complexity
[01:19](https://www.youtube.com/watch?v=n0sVot6s3-E&t=79s)
Whatnot began with a simple medallion architecture, then added systems for recommendations, in-app analytics, Kafka and other streaming integrations, experimentation, operational tooling, and real-time analytics. Each new business use case added more data products and more relationships between systems. Bailey says this success can leave a data platform team unsure who owns each component, which systems are still active, and which failures are affecting the wider platform.

### Data mesh provides principles, but a full implementation can be too heavy
[03:46](https://www.youtube.com/watch?v=n0sVot6s3-E&t=226s)
Bailey describes data mesh as an approach where teams create, share, and govern data products. Its principles include treating data as a product, providing a self-serve data platform, giving domains ownership of their data, and applying governance across the system. He also points out that centralization can be efficient early in an organization's life. Teams may have complex data needs without having enough organizational size to support fully decentralized ownership.

### Complex data platforms need consistent service patterns
[05:43](https://www.youtube.com/watch?v=n0sVot6s3-E&t=343s)
After deploying and later deprecating many systems, Whatnot found that it needed a repeatable way to stand up and monitor services. The pattern starts with naming the service and designing it for future self-service. Data engineering owns the interface end-to-end, including how users create and consume data. A central control plane provides shared practices for infrastructure, monitoring, and orchestration, while performance is measured through service-level objectives.

### The event bus hides implementation complexity behind a simple contract
[07:13](https://www.youtube.com/watch?v=n0sVot6s3-E&t=433s)
Whatnot's event bus lets backend, frontend, iOS, and Android developers create analytical events for consumption in the data warehouse, a real-time database, or Kafka-based application systems. The underlying setup includes producers, consumers, and a Protobuf schema registry. The business-facing service needs to make event production and consumption easy and define contracts between them. The other components remain implementation details.

### Producer and consumer interfaces need to fit the whole platform
[08:54](https://www.youtube.com/watch?v=n0sVot6s3-E&t=534s)
Whatnot built producer clients that understand the event bus rather than using only a Kafka client or Segment SDK. These wrappers connect application developers to the rest of the event system. The team also created simple consumer interfaces for company-wide use. Shared tools, including Dagster, DataDog, and Terraform, manage the event bus in the same way as other services, giving engineers common places to inspect orchestration, status, infrastructure, and SLOs.

### A simple pattern takes organizational agreement to enforce
[10:00](https://www.youtube.com/watch?v=n0sVot6s3-E&t=600s)
The event bus design looks simple from the outside, but Bailey says it required substantial work and consensus. A pattern must apply across many types of systems, while the team must also understand it, support it, and feel comfortable using it. The value comes from making the pattern enforceable and familiar rather than from adding another architectural diagram.

### Data teams still need modular deployment patterns
[10:52](https://www.youtube.com/watch?v=n0sVot6s3-E&t=652s)
Bailey says data mesh's organizational ideas are useful, but Whatnot still needed consistent patterns for deploying and managing data services. He compares the current state of data pipes and movement between systems to the period before Docker. The platform has compute and tooling, but lacks a modular way to package complexity so that data systems are easy to track and change.

## Notable quotes
- "Success breeds more complexity, which is great on one hand because that probably means that you're doing important things and impactful things for the business, but it also means as a data platforms team, it can be very easy to go underwater." (03:02)
- "You need to stretch the data engineering organization to own interfaces end-to-end." (06:53)
- "At the end of the day, what really matters to the business is that we can produce events, we can consume events, and we have a way to define them very easily that allows us to create contracts between consumers and producers." (08:13)
- "What it feels like on the ground to me is that we're not missing this organizational pattern so much as we're still missing good consistent patterns around deployment and management of data services." (10:52)

## Tools & references mentioned
- Whatnot
- Data Mesh
- Kafka
- Protobuf
- Segment SDK
- Dagster
- DataDog
- Terraform

## Who should watch
- You are adding analytics, machine learning, or real-time systems faster than your platform team can standardize how they are operated.
- Your organization is considering data mesh and needs a practical view of where centralization still makes sense.
- You want an example of how service contracts, shared tooling, and SLOs can make a data platform easier for application teams to use.

## Editor's note

Stephen Bailey says Whatnot needed repeatable patterns for deploying and managing data services as its platform grew more complex. ZenML lets teams write workflows as Python steps and run the same pipeline on different infrastructure through a configured stack. Each run records its steps, inputs, outputs, and code version, so teams can trace how an artifact was produced.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [The Future of ML and Data Platforms](https://mlopstalks.com/talks/the-future-of-ml-and-data-platforms) (Michael Del Balso, Tecton, 55:17)
- [The Only Constant is (Data) Change](https://mlopstalks.com/talks/the-only-constant-is-data-change) (Benjamin Rogojan, Seattle Data Guy & Chad Sanderson, Gable & Christophe Blefari, NAO & Maggie Hays, Acryl Data, 40:50)
- [Aggressively Helpful Platform Teams](https://mlopstalks.com/talks/aggressively-helpful-platform-teams) (Stefan Krawczyk, Stitch Fix, 51:52)
- [How Data Platforms Affect ML & AI](https://mlopstalks.com/talks/how-data-platforms-affect-ml-ai) (Jake Watson, The Oakland Group, 39:12)
- [Trustworthy Data for Machine Learning](https://mlopstalks.com/talks/trustworthy-data-for-machine-learning) (Chad Sanderson, Convoy, 51:04)