# On Structuring an ML Platform 1 Pizza Team

Breno Costa & Matheus Frata, Neoway | MLOps Coffee Sessions | Episode 73 | 52:42

Source: https://www.youtube.com/watch?v=66A72NgSfeE
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/on-structuring-an-ml-platform-1-pizza-team
Published: 2022-01-10
Tags: data-engineering, deployment, developer-experience, platform-teams

## TL;DR
- Neoway reorganized from functional silos into stream-aligned and platform teams because dependencies made it difficult to move models into production.
- The four-person machine learning platform team gives data scientists templates and services for feature engineering, pipelines, deployment, and monitoring.
- The team treats the platform as a product by interviewing users, mapping their problems across the model lifecycle, and prioritizing work around user value.

## Summary
Breno Costa and Matheus Frata describe how Neoway created a machine learning platform team during a wider organizational redesign. The company had previously grouped people into functional areas, which made data access, deployment, and software delivery depend on several other teams. Using ideas from Team Topologies and Conway's law, Neoway formed stream-aligned teams around business areas and platform teams that provide internal services. The machine learning platform team now supports data scientists with templates and tooling for exploration, feature engineering, model training, deployment, and monitoring. The guests explain how the machine learning and data platforms interact, why they kept technologies such as Airflow after testing alternatives, and how their feature store grew from an internal need. They are direct about the cost of building internal tools. The platform team first interviewed users, created personas and a story map, then prioritized the parts of the model lifecycle that would deliver value sooner.

## Key ideas
### Neoway's functional silos made model delivery depend on too many teams
[06:30](https://www.youtube.com/watch?v=66A72NgSfeE&t=390s)
Matheus Frata says Neoway originally organized people into functional areas, including data science, applications, and customer experience. The arrangement created communication problems. The data science group struggled to obtain historical data in a usable structure and to deploy models through the company's architecture. Breno Costa adds that data scientists, machine learning engineers, and data operations specialists worked in one team, while deployment still required conversations with two or three other teams, database changes, and configuration files. The result was slow delivery of business value and frustration when bottlenecks blocked the team.

### The redesign used team structure to support the architecture and business outcomes
[08:40](https://www.youtube.com/watch?v=66A72NgSfeE&t=520s)
Breno Costa connects Neoway's redesign to Team Topologies and Conway's law, the idea that software architecture reflects the communication structure of the organization. The company used a team-first approach and organized groups around the architecture and the outcomes they needed to deliver. Some teams focus on business value in stream-aligned cells. Others provide internal services through platform teams. Matheus Frata says the engineers were involved in decisions about the new structure, including how communication would work when deploying a model. That participation made the change easier to accept.

### Platform teams reduce technology complexity for stream-aligned teams
[16:14](https://www.youtube.com/watch?v=66A72NgSfeE&t=974s)
Matheus Frata explains that platform teams simplify difficult technology for stream-aligned teams. Examples include deploying a Kubernetes server, launching a Spark job, or accessing a data lake. The platform also helps prevent separate teams from rebuilding parallel solutions. Platform teams can take an enablement role when another group needs advice or mentoring. At Neoway, an engineering productivity team helped the machine learning platform create a base template for launching Spark infrastructure. Stream-aligned teams communicate with the platform through defined channels such as documentation and Jira tickets.

### The data platform provides shared services for machine learning
[18:44](https://www.youtube.com/watch?v=66A72NgSfeE&t=1124s)
Neoway's data platform handles data ingestion, storage, and retrieval. Matheus Frata describes a schema registry where data must be registered, which helps maintain backward compatibility. Other services include a data hub, a data lake, and a data API that exposes data through REST. The machine learning platform builds its tools on these services instead of recreating them. The feature store uses parts of the data platform, as do model deployment and machine learning pipelines. Unstructured legal data still runs on a mostly separate platform, although Neoway intends to integrate the two over time.

### Different teams need different interaction modes
[21:07](https://www.youtube.com/watch?v=66A72NgSfeE&t=1267s)
Breno Costa describes three interaction modes from Team Topologies: collaboration, facilitation, and X-as-a-service. Collaboration means teams work together for a defined period to discover and deliver something. Facilitation means one team helps another learn a topic. X-as-a-service means one team provides a service that another consumes, such as a self-service API. Neoway uses these modes according to the team's experience and the work involved. An experienced sales and marketing cell moved from collaboration to using the feature store SDK through documentation. A more junior team needed facilitation to understand feature stores and schemas.

### Templates moved model deployment closer to a self-service workflow
[26:13](https://www.youtube.com/watch?v=66A72NgSfeE&t=1573s)
Before the redesign, a data scientist produced a model repository and artifacts, then a data operations professional had to coordinate with other teams to put the model into production. After the redesign, the machine learning platform supplies tools and templates so data scientists can handle the steps from data exploration and training through deployment and monitoring. Breno Costa says the model templates include Python model structure, unit or integration testing, CI pipelines, Docker image publishing, and production deployment. Matheus Frata says deployment that could take two or three weeks can now happen through a single command for some use cases.

### A small platform team supports users across several cells
[31:59](https://www.youtube.com/watch?v=66A72NgSfeE&t=1919s)
Breno Costa says the machine learning platform has four technical members: Breno Costa, Matheus Frata, Manuel, and Victor. They work together to support roughly 15 data scientists in stream-aligned cells, along with data engineers and other users. The guests also describe a larger platform group that includes product and delivery roles. Their setup separates the team size from the number of people who use the platform. Demetrios Brinkmann points out that platform investments can support more users without adding people one for one, although the guests also describe the support work needed to keep services running.

### The technology stack evolved through experiments and existing platform capabilities
[33:42](https://www.youtube.com/watch?v=66A72NgSfeE&t=2022s)
Neoway moved to Google and tested technologies as its scale needs changed. The team used BigQuery, MongoDB, Airflow, Kubernetes, Spark, and Dataproc in different parts of the platform. Airflow stayed because it solved their scheduling needs even as newer libraries appeared. The team moved away from Jupyter Enterprise Gateway after maintenance and resource contention became difficult. With Dataproc, each data scientist could use a separate cluster, although the team had to remind users not to waste money. Breno Costa says the platform now has a foundation, so changes require testing and discussion about how they will integrate.

### Platform work starts with user problems rather than technical possibilities
[44:04](https://www.youtube.com/watch?v=66A72NgSfeE&t=2644s)
Matheus Frata says the team learned to frame the problem before building a solution. An earlier feature builder project attracted users, but the team had not thought enough about developer experience, so it became difficult for data scientists to add features. For the newer platform, they interviewed users and created personas for data scientists with different work patterns, data engineers, and people who work directly with customers. They mapped problems across feature engineering, exploratory analysis, deployment, and monitoring, then used that map to decide what to prioritize. Breno Costa adds that scalability and maintainability matter, but a platform can still fail if users do not adopt it.

## Notable quotes
- Breno Costa: "The platform solutions can't fail at the user adoption." (47:57)
- Breno Costa: "We need to put the user on the center, talk with user and listen to the user." (48:47)
- Breno Costa: "We are building our platform on the giant shoulders." (40:03)
- Matheus Frata: "We have to think about the data scientists using the stuff." (51:52)

## Tools & references mentioned
- Neoway
- MLOps Community
- Demetrios Brinkmann
- Vishnu Rachakonda
- Team Topologies
- Conway's law
- feature store
- Google
- Kubernetes
- Spark
- Airflow
- BigQuery
- MongoDB
- Dataproc
- Jupyter Enterprise Gateway
- Jira
- Docker

## Who should watch
- You are deciding whether an ML platform should be a separate team or part of a data science group, and need an example of the organizational trade-offs.
- Your data scientists depend on several teams to deploy models, and you want to see how templates, shared services, and defined team interactions can reduce that work.
- You are building internal ML tools and need a practical warning about developer experience, user interviews, support costs, and adoption.

## Editor's note

Breno Costa says Neoway's old functional silos left model deployment dependent on two or three other teams, database changes, and configuration files. ZenML lets teams write ML workflows as Python pipelines, then run the same pipeline code on different infrastructure, including a laptop, Kubernetes, Airflow, or a cloud provider's services.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [The Future of ML and Data Platforms](https://mlopstalks.com/talks/the-future-of-ml-and-data-platforms) (Michael Del Balso, Tecton, 55:17)
- [Lessons from Studying FAANG ML Systems](https://mlopstalks.com/talks/lessons-from-studying-faang-ml-systems) (Ernest Chan, Duo Security, 45:32)
- [How Data Platforms Affect ML & AI](https://mlopstalks.com/talks/how-data-platforms-affect-ml-ai) (Jake Watson, The Oakland Group, 39:12)
- [Building an ML Platform: Insights, Community, and Advocacy](https://mlopstalks.com/talks/building-an-ml-platform-insights-community-and-advocacy) (Stephen Batifol, Wolt, 45:49)
- [How to Leverage ML Tooling Ecosystem](https://mlopstalks.com/talks/how-to-leverage-ml-tooling-ecosystem) (Mariya Davydova, Neu.ro, 55:57)