# Scaling Machine Learning with Data Mesh

Shawn Kyzer, Thoughtworks | MLOps Coffee Sessions | Episode 116 | 53:54
Hosted by Adam Sroka

Source: https://www.youtube.com/watch?v=HebrADcRtZY
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/scaling-machine-learning-with-data-mesh
Published: 2022-08-17
Tags: data-engineering, feature-engineering, governance, platform-teams

## TL;DR
- Data Mesh needs organizational education and cultural change alongside its technical architecture.
- Machine learning data products can extend Data Mesh across analytics and operational planes while retaining discoverability, quality, security, and interoperability.
- Teams should begin with valuable, achievable use cases, establish sensible data and MLOps practices, then turn repeated machine learning patterns into self-service templates.

## Summary
Shawn Kyzer explains how Data Mesh principles can apply to machine learning projects. Data products began as curated, discoverable, trustworthy outputs for analytics, but machine learning crosses into operational systems when a trained model is exposed through an API. Shawn describes machine learning products as consumer-aligned, multi-plane products that may include feature extraction, model quality checks, discoverability, security, and reusable outputs. He argues that tools should come after use cases, domain boundaries, product design, and capability mapping. The conversation also covers exploratory work, where data scientists may need a well-governed, complete data set alongside business-facing views. Shawn says data scientists are consumers and stakeholders who can request specialized output ports. For implementation, he recommends a thin slice, a few high-value use cases, and templates that can eventually power a command-line or graphical self-service interface. The organizational side matters as much as the technology. A full Data Mesh needs cultural buy-in and a mandate for operating-model changes.

## Key ideas
### Data Mesh must change how the organization works, not only which tools it uses
[01:13](https://www.youtube.com/watch?v=HebrADcRtZY&t=73s)
Shawn says Data Mesh is often misunderstood as a collection of technologies. Its principles include discoverability, addressability, trustworthiness, and security, but those capabilities do not create the operating model by themselves. The organization has to learn how to work with domain-owned data products and federated governance. He draws a direct parallel with MLOps, where adopting tools does not resolve ownership, communication, or decision-making problems. Later, he says a technical implementation can still bring useful practices such as versioning, model quality checks, and loosely coupled pipelines, but it will miss much of the benefit without cultural buy-in and a mandate from senior leadership.

### Machine learning exposes a gap between analytics data products and operational systems
[05:17](https://www.youtube.com/watch?v=HebrADcRtZY&t=317s)
Shawn describes an early Data Mesh approach that worked well for analytics. Teams captured time-series events, divided them into meaningful products, and made them trustworthy and discoverable through catalogs, quality checks, and orchestration. Machine learning complicated this model because a project could begin in the analytics plane, where data was prepared and a model was trained, then move into the operational plane when an API was used for fraud detection or delivery estimates. The existing data-product patterns and tools did not map cleanly onto this movement across planes. Shawn says the answer is to extend the framework rather than treat machine learning as an exception.

### A machine learning data product can include features, models, and reusable operational outputs
[08:49](https://www.youtube.com/watch?v=HebrADcRtZY&t=529s)
Shawn places machine learning products within the idea of a multi-plane, consumer-aligned data product. Feature extraction or feature engineering may sit beside the main product. The same product principles still apply: a model should be trustworthy, its drift should be measured, and it should be discoverable for teams that want to reuse an algorithm or transfer knowledge. He describes applying those principles to the product's capabilities, including discoverability, addressability, trustworthiness, and security. The product may connect source-aligned data products to aggregates, a feature store, a trained model, and an API or other output used by an operational system.

### Data scientists need both convenient views and governed access to more complete data
[11:27](https://www.youtube.com/watch?v=HebrADcRtZY&t=687s)
Adam asks how exploratory modeling works when a Data Mesh presents carefully curated products and the required signal is not yet known. Shawn gives the example of a source-aligned event product whose business-analyst output removes much of the original detail. A business user may need 15 columns, while a data scientist may need to inspect 200 columns to decide whether a field could become a feature. His solution is to provide another output port with a more complete, well-catalogued data set. It is not an unmanaged raw dump. The data still has measured quality, while the interface is shaped for exploration and model development.

### Data scientists are stakeholders who can request specialized output ports
[14:40](https://www.youtube.com/watch?v=HebrADcRtZY&t=880s)
When Adam worries about pushing feature requests upstream, Shawn answers that a data scientist is a consumer and an internal stakeholder. If the data is needed in a particular form, the consumer can request a specialized output port from the domain that owns the product. Shawn also questions why MLOps and data science teams must remain fully centralized. He suggests that machine learning engineers and data scientists could work in cross-functional teams aligned to domains or particular data products. This keeps ownership closer to the use case, although Adam points out that staffing and scarce skills can make that model difficult.

### Technology choices should follow use cases and required capabilities
[17:53](https://www.youtube.com/watch?v=HebrADcRtZY&t=1073s)
Shawn starts architecture work with high-value use cases, then extracts domains and divides the work into meaningful data products. He maps how products feed one another, such as source-aligned systems feeding an aggregate that supplies a feature store. Only after that does he map abstract capabilities such as schema management, data quality, and storage to technologies. Existing tools should remain when they fit the need and the organization already uses them. A team without infrastructure skills may need a managed service, while a larger engineering group may choose open source and operate Kubernetes or a customized Kubeflow deployment. He avoids a universal product recommendation because each organization's constraints differ.

### A thin slice can turn repeated machine learning patterns into self-service
[23:29](https://www.youtube.com/watch?v=HebrADcRtZY&t=1409s)
For teams building a platform for software engineers who need to deploy standard machine learning models, Shawn recommends starting with a small set of mature, usable data products. The team builds one complete multi-plane machine learning product, then identifies repeated components and turns them into blueprints or templates. A command-line interface can provision infrastructure, create a pipeline skeleton, and commit the result to a repository. After several patterns work, a team can add a graphical interface, such as a Backstage setup. Shawn says the first implementation is slow, but the templates become more useful as common patterns are identified. He calls the initial end-to-end implementation a thin slice.

### Early progress comes from simple, valuable use cases and agreed fitness functions
[30:37](https://www.youtube.com/watch?v=HebrADcRtZY&t=1837s)
Shawn recommends ranking possible use cases by business value and difficulty. The first case should be achievable within roughly three to six months, so the team can show value without cutting corners on a complex project. Success measures depend on the organization. A developer-focused group might measure whether the developer experience improves or whether teams stop rebuilding similar architectures. A business-focused organization might track how often users access a data product's output ports. Catalog search and usage analytics can also show whether people are finding and using the products. The measures need to be agreed before implementation.

### Data Mesh is a poor fit for small, simple products until growth creates real bottlenecks
[39:13](https://www.youtube.com/watch?v=HebrADcRtZY&t=2353s)
Shawn does not recommend Data Mesh for every company. A small startup with a simple product and a need to move quickly may gain more from a monolith than from a distributed network of independently managed products. Data Mesh adds coordination and interoperability work. The case changes when a growing company hits bottlenecks in its warehouse, streaming systems, or transaction workloads. At that point, decomposing the data monolith into smaller products can make the system easier to manage. Shawn also says that good data stewardship, testing, automation, and other MLOps practices are useful preparation before an organization attempts the broader cultural and architectural change.

## Notable quotes
- Shawn Kyzer: "You don't get nearly as much benefit as if you also applied the operating model." (42:20)
- Shawn Kyzer: "You are a consumer. At the end of the day you're a stakeholder." (14:40)
- Shawn Kyzer: "The techy stuff or the tools, it's the last thing that I do." (17:53)
- Shawn Kyzer: "You have to start with that first one." (24:26)
- Shawn Kyzer: "At least now you know that the source data that you're getting has some kind of quality check on it." (36:55)

## Tools & references mentioned
- Thoughtworks
- Data Mesh
- MLOps
- Zhamak Dehghani
- Great Expectations
- Airflow
- Luigi
- Kubeflow
- Delta Lake
- Presto
- Trino
- SageMaker
- Kubernetes
- MLflow
- TensorFlow
- Prometheus
- Grafana
- DVC
- Redshift
- Backstage
- Google Analytics
- Spotify
- Lexicon
- ODSC

## Who should watch
- You are designing a machine learning platform and need to decide how data products should connect to features, models, and operational APIs.
- Your data science team depends on poorly documented data and you need a practical way to support exploration without abandoning quality controls.
- Your organization is considering Data Mesh and needs to judge whether its size, use cases, skills, and culture justify the added coordination.

## Editor's note

Shawn Kyzer says repeated machine learning patterns should become self-service templates after a team has built a complete, thin-slice product. ZenML lets teams write ML workflows as Python pipelines whose runs record steps, inputs, outputs, and code version, so a template carries a traceable execution history.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [Data Mesh: Data Quality Control Mechanism for MLOps?](https://mlopstalks.com/talks/data-mesh-data-quality-control-mechanism-for-mlops) (Scott Hirleman, DataStax, 57:03)
- [How Data Platforms Affect ML & AI](https://mlopstalks.com/talks/how-data-platforms-affect-ml-ai) (Jake Watson, The Oakland Group, 39:12)
- [A Journey in Scaling ML](https://mlopstalks.com/talks/a-journey-in-scaling-ml) (Gabriel Straub, Ocado Technology, 52:41)
- [Data Engineering for ML](https://mlopstalks.com/talks/data-engineering-for-ml) (Chad Sanderson, Convoy, 57:54)
- [Exploring SQLmesh](https://mlopstalks.com/talks/exploring-sqlmesh) (, 1:18:58)