Jake Watson argues that data platforms range from a simple database and transformation flow to complex systems with custom software and many teams.
2
Data mesh tends to fit large, distributed organisations better than companies with a central data team that already works well.
3
Machine learning teams should understand existing business logic and analytics before reaching for raw data or building new features.
Summary
Jake Watson explains how data platforms support machine learning and AI work. He describes platforms as systems that can be as small as a database with transformations and outputs, or as complex as the custom platforms built by companies such as Uber and Airbnb. The hard part of scaling is often organisational. Data mesh can help large, distributed organisations, but it is a slow process and does not suit every company. Watson recommends choosing data models for the use case instead of forcing one model across machine learning, business intelligence, and log analytics. He also connects data quality to technical debt, since tests cost money and poor quality makes future change harder. For ML teams, a trusted descriptive analytics foundation gives them a better starting point. Watson warns that raw data access can create political and practical problems, and that teams may waste time rebuilding business logic that already exists in analytics work.
Watson says a data platform may be a database that performs transformations and produces outputs, or it may contain many vendors and custom software. His consultancy often starts with clients that have a blank sheet of paper and need help gathering requirements and choosing an architecture. Building a platform is easier than keeping it running in production, because maintenance needs a broad support network inside the company. He has worked with organisations that have no data team, small teams, and large teams with many microservices moving data around.
Scaling the people is harder than scaling the technology
Watson says a typical data team can start as a small group, but scaling beyond that becomes difficult because responsibilities need to move into separate teams. Those teams need enough autonomy to work at speed, while the organisation still needs shared governance. He connects this organisational problem with data mesh. In his view, some successful examples were already operating in a mesh-like way without using that name, with business lines owning platforms and a central governance layer managing shared concerns.
Watson does not present data mesh as a universal answer. He says it is more likely to help a large organisation that is naturally distributed, such as a global company or an insurer with separate home, car, and other lines of business. A company with a central data team that already works well may gain little from adopting it. He suggests trying domain-oriented data marts and other smaller changes first, then checking whether the original scaling problems remain before moving to a fully distributed structure.
Watson recommends flexibility in data modeling. Machine learning may need a wide table that can be passed into scikit-learn, PyTorch, or TensorFlow. Business intelligence tools such as Power BI and Tableau may work better with a star schema and dimensional modeling. Log analytics may use JSON in Elasticsearch or another system. Trying to make every workload use one model can create friction, so the model should fit the use case and the tools involved.
Watson says organisations are again discussing enterprise and conceptual modeling, including shared definitions for things such as customers and products. A common definition can reduce the risk of producing many incompatible versions of the same table. He is also wary of spending years trying to create one perfect customer table. His recommendation is to bring the useful parts of older modeling practices into cloud and agile work, while accepting some compromises so teams can deliver value.
Data quality is a cost and a form of technical debt
Watson is skeptical that teams can solve data quality simply by writing enough tests. Data quality checks query warehouses, and systems such as Databricks and Snowflake can charge for those queries. The tests also need maintenance. At the same time, weak quality makes a platform harder to change because teams lose confidence in their data. He compares accumulated data quality problems with technical debt: the cost appears later through slower changes, errors, and business users who stop trusting the platform.
The right order depends on the risk and delivery goal
For a quick proof of concept, Watson says a team may accept less modeling and fewer tests while it checks whether an idea can deliver value. Critical reporting, accounting, or safety work needs stronger quality controls. A system expected to grow also benefits from early modeling decisions. He recommends watching error rates, user trust, performance, and the speed of change. Those signals can show whether the team needs more quality work or needs to remodel data that has become a constraint.
ML teams should build on trusted analytics and existing business logic
Watson says ML work is easier when pipelines are known to work and the input data has an acceptable level of quality. He describes this as understanding historical data before forecasting or making predictions. Separate data science teams can move quickly and gain autonomy, but they also risk producing results from a weaker understanding of the data. He advises teams to examine existing analytics and business logic before using raw data, since many useful ML features may already exist in the form of business KPIs.
Pipeline tools make different tradeoffs for ML workloads
Watson says Airflow can become difficult to manage when ML workflows need feedback loops or repeated model-related steps. Directed acyclic graphs work best when the flow moves in one direction. He sees Prefect as more flexible for some ML work and mentions Kubeflow and other newer tools. Airflow can still be sufficient when the pipeline is largely one-way and the number and type of models remain manageable. The choice depends on the workload rather than on a single universally correct tool.
