Data and ML platforms should start with simple tools, then add shared capabilities after teams show repeated needs.
2
Operational ML needs to connect modern data workflows with production concerns such as low latency, monitoring, serving, and data consistency.
3
Platform teams work better when they combine centralized reporting with backlogs shaped by the teams they support.
Summary
Michael Del Balso and Erik Bernhardsson discuss how data platforms, ML systems, and software engineering are converging. They argue that companies should avoid building large internal platforms before understanding what their teams need. A small platform can support an initial use case, while repeated bespoke work reveals what belongs in the shared system. They compare centralized and decentralized team structures and settle on a hybrid approach, with central reporting and day-to-day work embedded in product teams. The conversation then turns to operational ML. Michael explains why production ML needs more than warehouse SQL, since features may need batch computation, real-time computation, historical reconstruction, low-latency serving, and consistent training and serving data. Erik describes how Spotify used nightly jobs for recommendations and why that worked for a problem where preferences changed slowly. Both guests place ML tooling on top of the data stack rather than in a completely separate system.
The modern data stack is fragmented enough to make adoption harder
Erik Bernhardsson describes data as a fragmented space with many widgets, layers, and frequent claims about a new data stack. Michael Del Balso adds that continually splitting systems into more layers can leave a new user wondering whether they need ten or twelve separate tools. They see room for simpler onboarding and perhaps more bundling, although Erik does not predict how much consolidation will happen. Erik also says the ML space is even less settled than the data space. Its layers are still forming, while the data ecosystem has had more time to mature.
SQL handles much of analytics, while ML still needs ordinary code
Erik says warehouses are better than Hadoop for most data work and that SQL is a sensible way to push filtering, joins, and grouping into the execution engine. He worries that the move toward SQL may have gone too far because some work still needs code. Michael says Tecton lets users combine SQL transformations with Python and data frame operations. He has seen little adoption of workflows that try to do ML entirely in SQL. Erik points to Snowpark as evidence that warehouse companies also see a need for data frame APIs and better support for data scientists.
Erik defines the reason to build an internal platform in terms of scale. Once enough people can become more effective through shared tools, the investment can pay for itself. He favors self-service access, such as giving people SQL access and making data available, then watching what they do and automating repeated work. Michael describes the same pattern from Uber's Michelangelo platform. The team built a basic system, helped teams build ML applications on it, and turned repeated bespoke work into platform features. Both warn against a central team building infrastructure far ahead of actual requirements.
Central reporting and embedded work avoid the worst team structures
Erik describes the problems at both extremes. Full centralization can leave business teams waiting for help and asking to hire their own people. Full decentralization can produce inconsistent skills, hiring standards, and data practices across the company. His preferred model has a centralized data team with decentralized backlog management. People report through a central data organization while working day to day with other teams. Michael describes a similar model for ML, but says Uber learned that an applied ML group must choose projects with a clear long-term owner. A use case needs a capable team, operational knowledge, and enough business importance for that team to take it over.
A platform team needs a service mentality and outcome measures
Erik says platform teams can fail even when they build good technology if other teams do not use it or value it. He suggests measuring platform teams partly through internal NPS and asking whether users get value and feel heard. Michael says Uber also struggled to determine the right team size and to connect platform work to supported business value. ML made that connection easier in cases such as surge pricing, while generic data platforms can sit farther from the final use of the data. Erik argues that executive support for data-driven decisions is an early requirement because the organization must agree to measure business impact, rather than celebrate feature launches alone.
Operational ML connects data workflows to production systems
Michael says Tecton focuses on operational ML, meaning ML that affects the production user experience. He does not want a separate ML stack disconnected from the main data platform. Instead, the ML system should use existing platforms such as Databricks or Snowflake while adding interfaces for feature transformations, training, serving, and production operations. Some features can be precomputed in batch, while others depend on real-time inputs such as a search query. The system must also preserve historical feature values so teams can build accurate training data. A major requirement is consistency between the data used in training and the data available when serving predictions.
Latency creates a boundary between analytics and production ML
Erik says training a model and deploying it can be relatively easy compared with generating and updating features in real time. Requirements change sharply when a system must respond in milliseconds instead of seconds, minutes, or hours. Spotify's recommendation system used nightly jobs and Cassandra because listeners' preferences did not change dramatically from one day to the next. That approach would not fit every product, especially systems that personalize an onboarding flow using a user's current context. Michael describes Tecton's scope as a focused attempt to handle feature data flows and their operational concerns without claiming to be a complete ML platform.
