Whatnot found that reducing data complexity requires consistent patterns for deploying, monitoring, and managing every kind of data service.
2
A data service should expose a simple interface, while its implementation details remain behind contracts, guardrails, and a shared control plane.
3
Data mesh offers useful principles, but Stephen Bailey argues that teams also need practical deployment and management patterns for data services.
Summary
Stephen Bailey describes Whatnot's effort to scale its data platform as the company added analytics, machine learning, real-time systems, and application use cases. A simple medallion architecture gradually became a complex collection of warehouses, streaming systems, experimentation tools, operational integrations, and real-time databases. Bailey uses data mesh as a useful reference, especially its ideas about data products, self-service platforms, domain ownership, and system-level governance. He also explains why full decentralization can be too heavy for a small or rapidly changing organization. Whatnot's response is to create repeatable patterns for data services. These patterns include clear service names, self-service interfaces with guardrails, end-to-end ownership of interfaces, a central control plane, shared infrastructure practices, monitoring, and service-level objectives. The event bus illustrates the approach: producers and consumers interact through simple interfaces and Protobuf contracts, while Dagster, DataDog, and Terraform provide common management tools. Bailey argues that data infrastructure is still in a pre-Docker stage, with too little modularity around data movement and pipelines.
Whatnot's growing data stack created operational complexity
Whatnot began with a simple medallion architecture, then added systems for recommendations, in-app analytics, Kafka and other streaming integrations, experimentation, operational tooling, and real-time analytics. Each new business use case added more data products and more relationships between systems. Bailey says this success can leave a data platform team unsure who owns each component, which systems are still active, and which failures are affecting the wider platform.
Data mesh provides principles, but a full implementation can be too heavy
Bailey describes data mesh as an approach where teams create, share, and govern data products. Its principles include treating data as a product, providing a self-serve data platform, giving domains ownership of their data, and applying governance across the system. He also points out that centralization can be efficient early in an organization's life. Teams may have complex data needs without having enough organizational size to support fully decentralized ownership.
Complex data platforms need consistent service patterns
After deploying and later deprecating many systems, Whatnot found that it needed a repeatable way to stand up and monitor services. The pattern starts with naming the service and designing it for future self-service. Data engineering owns the interface end-to-end, including how users create and consume data. A central control plane provides shared practices for infrastructure, monitoring, and orchestration, while performance is measured through service-level objectives.
The event bus hides implementation complexity behind a simple contract
Whatnot's event bus lets backend, frontend, iOS, and Android developers create analytical events for consumption in the data warehouse, a real-time database, or Kafka-based application systems. The underlying setup includes producers, consumers, and a Protobuf schema registry. The business-facing service needs to make event production and consumption easy and define contracts between them. The other components remain implementation details.
Producer and consumer interfaces need to fit the whole platform
Whatnot built producer clients that understand the event bus rather than using only a Kafka client or Segment SDK. These wrappers connect application developers to the rest of the event system. The team also created simple consumer interfaces for company-wide use. Shared tools, including Dagster, DataDog, and Terraform, manage the event bus in the same way as other services, giving engineers common places to inspect orchestration, status, infrastructure, and SLOs.
A simple pattern takes organizational agreement to enforce
The event bus design looks simple from the outside, but Bailey says it required substantial work and consensus. A pattern must apply across many types of systems, while the team must also understand it, support it, and feel comfortable using it. The value comes from making the pattern enforceable and familiar rather than from adding another architectural diagram.
Bailey says data mesh's organizational ideas are useful, but Whatnot still needed consistent patterns for deploying and managing data services. He compares the current state of data pipes and movement between systems to the period before Docker. The platform has compute and tooling, but lacks a modular way to package complexity so that data systems are easy to track and change.
"Success breeds more complexity, which is great on one hand because that probably means that you're doing important things and impactful things for the business, but it also means as a data platforms team, it can be very easy to go underwater."03:02
Who should watch
You are adding analytics, machine learning, or real-time systems faster than your platform team can standardize how they are operated.
Your organization is considering data mesh and needs a practical view of where centralization still makes sense.
You want an example of how service contracts, shared tooling, and SLOs can make a data platform easier for application teams to use.
