# Centralized or Decentralized ML Platform?

Jelmer Borst & Daniela Solis, Picnic | MLOps Podcast | Episode 267 | 57:50
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=DjqTkVASLC4
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/centralized-or-decentralized-ml-platform
Published: 2024-10-08
Tags: data-engineering, engineering-culture, monitoring, platform-teams

## TL;DR
- Picnic moved from a small centralized machine learning team to a hybrid model, then returned to a more centralized structure split across domain teams.
- Picnic's clean, centralized data platform reduced repeated data preparation and let machine learning engineers spend more time experimenting and building models.
- Machine learning teams own their systems end to end, including deployment, monitoring, fallbacks, and the business result produced in production.

## Summary
Jelmer Borst and Daniela Solis explain how Picnic organized machine learning as its use cases and model count grew. The company began with forecasting and expanded into delivery operations, personalization, search, recommendations, and supply chain decisions. Picnic first tried embedding some machine learning work in product teams, but model performance declined, knowledge sharing weakened, and several people left. The company returned to a centralized structure with separate teams covering broad domains, supported by a platform team. The guests also describe the technical choices behind this approach. Picnic built on its existing data and software infrastructure, added components as needed, and used lightweight internal tools rather than adopting every specialized platform early. Their engineers own the full path from experimentation through production, monitoring, fallbacks, and business outcomes. They argue that the right organizational model depends on company size, margins, use cases, and the trade-offs a team can accept.

## Key ideas
### Picnic's machine learning work expanded from forecasting into many operational and customer-facing decisions
[04:06](https://www.youtube.com/watch?v=DjqTkVASLC4&t=246s)
Jelmer Borst describes Picnic as an online supermarket with very low margins, where accurate purchasing forecasts matter. The first models predicted how many items to buy, then machine learning spread into delivery planning, route optimization, stopping time, parking time, search, recommendations, personalization, and meal suggestions. As the number of use cases grew, Picnic had to decide how to run models at scale while keeping development fast and preserving shared learning. The organizational question became as important as the modeling work. Jelmer says the company initially favored a centralized structure because data scientists could learn from peers and challenge each other's work.

### Picnic's centralized data platform reduced repeated data-cleaning work
[16:37](https://www.youtube.com/watch?v=DjqTkVASLC4&t=997s)
Jelmer says Picnic had around 150 analysts and data users, so asking each person to clean and interpret raw data would create duplicated effort. A centralized data engineering team connected customer, app, delivery, and supply-chain events. The team collected data from around 500 sources, cleaned it, and exposed it for use across the business. Daniela says this was especially valuable for machine learning because engineers did not have to spend most of their time discovering how the data was structured. Picnic started with a local PostgreSQL version, moved to Redshift, and later ran the platform on Snowflake.

### Picnic added machine learning infrastructure only when repeated pain created demand
[21:05](https://www.youtube.com/watch?v=DjqTkVASLC4&t=1265s)
Daniela explains that Picnic built on infrastructure already used by its technology teams and added machine learning components as the work became more complex. The company used Kubernetes for jobs and services, MLflow for experiment tracking, and a lightweight model registry built on Amazon S3. Jelmer says they avoided generalizing too early. Teams were allowed to build their own versions of a solution, even though that caused some duplication. Once several engineers encountered the same scaling problem, the pattern became clear enough to justify a shared component. This pull-based approach also made adoption easier because the platform addressed an existing problem.

### Full-stack ownership includes business outcomes, safeguards, and fallback behavior
[21:14](https://www.youtube.com/watch?v=DjqTkVASLC4&t=1274s)
Daniela says Picnic's machine learning engineers own the entire solution, including deployment and monitoring. Their responsibility does not stop when a model begins serving predictions. They also ask whether the model delivers the business value that justified building it, what happens when it cannot predict, and which heuristic or other model can be used as a fallback. Picnic monitors predictions against actual outcomes when those become available, checks that new predictions are not far from earlier ones, and verifies that predictions exist at all. The team also works with business analysts to turn domain knowledge into automated tests.

### Picnic's embedded experiment weakened model performance and knowledge sharing
[40:47](https://www.youtube.com/watch?v=DjqTkVASLC4&t=2447s)
When Picnic grew from around 10 machine learning engineers toward 18, it tried a hybrid structure. Some people remained centralized while others were embedded in product teams responsible for major use cases. Jelmer says the arrangement looked attractive because a product team had direct access to the people working on its problem. In practice, the models began performing worse over time, the team could not improve them as well as before, and multiple people left the company. The embedded setup also isolated engineers from peers with related skills. For Picnic, the business results showed that the apparent alignment benefit did not offset the loss of shared expertise.

### Picnic returned to centralized domain teams instead of one undivided central group
[43:13](https://www.youtube.com/watch?v=DjqTkVASLC4&t=2593s)
Picnic ultimately returned to a more centralized organization, divided into multiple teams. One team covers consumer work such as personalization and recommendations. Another covers supply-chain and operational use cases. A separate platform team supports them. Each domain team contains machine learning engineers who work across several models rather than one person being isolated inside a product team. Jelmer expects the structure to become more fine-grained as the company grows, but he does not want to split teams prematurely. Too many small teams create more coordination and alignment work, while one large team can become difficult to manage.

### The right structure depends on the company's economics and the level of performance it needs
[51:55](https://www.youtube.com/watch?v=DjqTkVASLC4&t=3115s)
Jelmer says there is no organizational model that fits every company. A business with high margins may get enough value from a reasonably good forecast or recommendation. Picnic's low-margin supermarket model leaves less room for mediocre performance, so it needs to keep improving models and operating efficiently. He also says the trade-offs of embedding are often missing from public engineering blogs, which usually describe successful changes rather than failures. Teams choosing a structure should examine what they gain in local alignment and what they risk losing in knowledge sharing, innovation, and model performance.

### Hiring fewer people more carefully supports end-to-end machine learning work
[54:02](https://www.youtube.com/watch?v=DjqTkVASLC4&t=3242s)
Picnic looks for people who combine machine learning knowledge with software engineering, analytical thinking, business understanding, and willingness to take ownership when systems fail. Jelmer describes these candidates as difficult to find and says the company prefers waiting longer to hire the right person. Daniela adds that shared knowledge helped the team keep up with state-of-the-art methods when those methods were useful, without adopting them just because they were fashionable. The guests connect this hiring approach to centralized teams, where engineers can learn from colleagues and maintain a high standard across different use cases.

## Notable quotes
- Daniela Solis: "The boundary doesn't stop when the model is live and running." (13:02)
- Jelmer Borst: "You might sometimes slow things down to go faster in the future." (19:56)
- Jelmer Borst: "We'd rather see a bit of duplication happening in a few places and then you definitely see all the different patterns and the different options and struggles." (24:58)
- Jelmer Borst: "The actual business impact was actually really lacking." (45:26)
- Daniela Solis: "I do think that there's no unique model that helps for every company." (52:40)

## Tools & references mentioned
- Picnic
- MLOps Community
- DoorDash
- Netflix
- PostgreSQL
- Redshift
- Snowflake
- Kubernetes
- MLflow
- Amazon S3
- dbt

## Who should watch
- You are deciding whether machine learning practitioners should sit in product teams or learn together in a central organization.
- Your team has many models in production and needs to connect model monitoring with business outcomes, fallbacks, and operational ownership.
- You are building an internal ML platform and want to add shared tools when repeated team pain creates demand instead of standardizing everything in advance.

## Editor's note

Daniela Solis says Picnic added machine learning infrastructure only when repeated pain created demand, rather than generalizing too early. ZenML lets teams write Python pipelines while configuring the infrastructure underneath, giving a platform team a shared path without forcing every use case into the same setup.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [On Structuring an ML Platform 1 Pizza Team](https://mlopstalks.com/talks/on-structuring-an-ml-platform-1-pizza-team) (Breno Costa & Matheus Frata, Neoway, 52:42)
- [Platform Thinking: A Lemonade Case Study](https://mlopstalks.com/talks/platform-thinking-a-lemonade-case-study) (Orr Shilon, Lemonade, 51:42)
- [Building ML/Data Platform on Top of Kubernetes](https://mlopstalks.com/talks/building-ml-data-platform-on-top-of-kubernetes) (Julien Bisconti, 48:13)
- [Lessons from Studying FAANG ML Systems](https://mlopstalks.com/talks/lessons-from-studying-faang-ml-systems) (Ernest Chan, Duo Security, 45:32)
- [Building an ML Platform at SurveyMonkey](https://mlopstalks.com/talks/building-an-ml-platform-at-surveymonkey) (Shubhi Jain, SurveyMonkey, 55:42)