# Building ML/Data Platform on Top of Kubernetes

Julien Bisconti | MLOps Coffee Sessions | Episode 86 | 48:13
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=u1ggSj0OwMU
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/building-ml-data-platform-on-top-of-kubernetes
Published: 2022-03-12
Tags: build-vs-buy, cost, platform-teams, reliability

## TL;DR
- Julien Bisconti argues that organizational problems usually take more work to solve than technical problems, because changing processes requires people to understand context and trade-offs.
- Building an ML platform from scratch can take about two years, while the cloud continues to change during that time and adds maintenance, training, on-call, security, and recovery costs.
- ML engineers should receive an abstraction that lets them focus on machine learning, while platform teams use data, standardization, and documented decisions to decide what to build.

## Summary
Julien Bisconti discusses the practical work behind building an ML and data platform on Kubernetes. He argues that code is often easier to change than organizational processes, since teams need shared context and a clear reason for changing how they work. A platform built from scratch can take about two years, and the organization must then maintain it, train people, operate it, monitor it, and pay for it while cloud services continue to evolve. Julien recommends starting with cloud services, learning the problem and the right abstraction level, then building only when a specific business need justifies it. He also explains chaos engineering as a way to reveal existing system failures through hypotheses and controlled experiments. For ML systems, recovery is complicated because failures involve data, models, and code. His advice is to standardize internal interfaces, keep systems simple, document architectural decisions, and give ML engineers tools that hide infrastructure details they do not need to manage.

## Key ideas
### Organizational processes are harder to change than code
[04:17](https://www.youtube.com/watch?v=u1ggSj0OwMU&t=257s)
Julien Bisconti says his site reliability work covers reliability, cloud cost, developer platforms, and observability. He has found that most problems are organizational first. Code can be changed directly, while a process requires people to understand why it exists, what trade-offs it makes, and how it should work. He also says human communication has limited bandwidth, especially when teams work remotely. Data helps organizations make decisions faster and make more decisions over time. For Julien, speed of execution is the only consistent measure of company success he has found.

### Building an ML platform from scratch carries a large ownership cost
[07:51](https://www.youtube.com/watch?v=u1ggSj0OwMU&t=471s)
Julien estimates that building a machine learning platform from scratch takes about two years, assuming the team already knows what it is doing. He compares this with early cloud decisions, where companies often compared virtual machine prices with the cost of buying servers while ignoring the data center, internet connection, and telecom costs. An internal platform also requires finding the right API abstractions, teaching users, creating on-call rotations, and operating the system. During the build, cloud services keep changing, so a team can finish with a platform that is difficult to maintain and already behind available services.

### Career growth depends on what a person wants to optimize
[15:12](https://www.youtube.com/watch?v=u1ggSj0OwMU&t=912s)
When Demetrios Brinkmann asks whether engineers should build systems from first principles for their resumes, Julien recommends thinking about the outcome they want from their career. He says negotiation training may produce a higher return than another technical credential because it affects compensation directly. Teaching, talks, and meetups helped him build professional relationships after moving to Sweden. He is skeptical that building a database or open-source project automatically improves a career, since production software has a separate burden of maintenance and support. Julien says helping other people has brought him more satisfaction than collecting impressive technical skills.

### Chaos engineering tests hypotheses about existing failure modes
[22:01](https://www.youtube.com/watch?v=u1ggSj0OwMU&t=1321s)
Julien explains chaos engineering through service-level objectives and error budgets. An error budget gives a team room to run experiments when its service is performing within an acceptable range. He says the term is misleading because the team does not add chaos. It reveals chaos already inside the system. An experiment starts with a hypothesis, such as newly created machines being removed first during scale-down, and then tests that hypothesis. Monitoring must already be good enough to show what happened. The experiment can expose system behavior and edge cases that nobody had previously understood.

### ML failure recovery is harder because systems contain data, models, and code
[27:13](https://www.youtube.com/watch?v=u1ggSj0OwMU&t=1633s)
For a model that behaves badly in production, a team might roll back to the previous model or fall back to a heuristic. Julien points out that the previous model must still exist and be easy to deploy. Conventional operations tools are designed around code, while ML systems also handle data at much larger sizes. A load balancer does not automatically know that a failed model should trigger deployment of an earlier version. Running two models in production also complicates monitoring because the team must distinguish which model served each request. These details make ML recovery difficult to design.

### Platforms should hide infrastructure work from ML engineers
[29:08](https://www.youtube.com/watch?v=u1ggSj0OwMU&t=1748s)
Julien questions whether ML engineers should be expected to manage both machine learning and operations. He recalls data scientists asking what Docker was and says he had to teach them the basics. A data scientist may not want to spend the day debugging Ubuntu, GPUs, and drivers. The platform team's role is to provide the right abstraction, then adjust it through conversation and iteration. Julien connects this to psychological safety, where people can discuss gaps in their knowledge without being punished. That makes it easier to learn what users actually need from the platform.

### Use the cloud to learn the problem before building your own platform
[33:36](https://www.youtube.com/watch?v=u1ggSj0OwMU&t=2016s)
Julien recommends using cloud services because they let a team learn what its use case and abstraction should look like before rebuilding the system. He suggests considering whether ML is even needed for the whole problem, or whether a simpler approach can cover part of the use case. Paid services may cost less than the time and people needed to recreate them. Once a specific limitation becomes important to the business, building a missing platform component becomes easier to justify. The same need may also point toward an open-source project or a product for other companies.

### Standardization and simple documented decisions reduce operational mistakes
[36:24](https://www.youtube.com/watch?v=u1ggSj0OwMU&t=2184s)
Julien uses an example from airplane safety research to explain why consistent interfaces matter. Standardizing the cockpit dashboard removed many errors caused by pilots encountering different layouts. He applies the same idea inside companies: one language, platform, and API can make it easier for people to know where to look and what to use. He also recommends documenting architectural decisions in a short page that records the problem, the proposed solution, and the possibility of being wrong. He argues that simple systems, runbooks, and deliberate risk decisions can be better than automating every rare operation.

### Multi-cloud is mainly a negotiation strategy rather than a reliability plan
[43:15](https://www.youtube.com/watch?v=u1ggSj0OwMU&t=2595s)
Julien says CPU, memory, and networking are broadly similar across cloud providers, while security models differ and are difficult to translate. This makes multi-cloud technically expensive. He sees a practical use for multi-cloud when it gives a company negotiating power during contract renewal. He does not recommend treating it as an easy reliability solution. Migration can be so difficult that some teams allow an old project to die and rebuild it in the new cloud. His broader advice is to make infrastructure decisions according to business needs, cost, and the risks the organization can accept.

## Notable quotes
- Julien Bisconti: "Most problems are truly organizational first, and I would say that it's much harder to change a process than it is to change code." (04:40)
- Julien Bisconti: "When you start building an ML platform you start from nothing, and let's say that two years later you have your platform, but during those two years the cloud didn't freeze, it still evolved." (10:26)
- Julien Bisconti: "We don't add chaos to the system, we reveal the chaos that is already inside the system." (22:27)
- Julien Bisconti: "If I'm a data scientist and suddenly I'm a glorified sysadmin doing Ubuntu debugging with GPU and drivers and all those things, why should they hate themselves? I would hate myself if it was my job." (29:57)
- Julien Bisconti: "Don't ask the barber if you need a haircut. If you ask a developer what's the problem, he's going to come up with a code solution." (46:25)

## Tools & references mentioned
- Kubernetes
- Spotify
- Superwise
- Google Cloud
- Uber
- Docker
- Spanner
- Vertex AI
- SageMaker
- Simon Wardley
- Crossing the River by Feeling the Stones
- LaunchDarkly
- Walmart Labs
- ZenML
- StockX

## Who should watch
- You are deciding whether to build an internal ML platform or use cloud services and need to account for operating work beyond the initial implementation.
- Your ML team is spending time on containers, GPUs, deployment, access, or recovery instead of model and data problems.
- You want a practical explanation of chaos engineering for systems where data, models, infrastructure, and business processes can all fail.

## Editor's note

Julien Bisconti says a platform built from scratch can take about two years and then carry ongoing maintenance, training, and on-call costs. ZenML lets teams write ML workflows as Python pipelines while choosing the orchestrator and infrastructure through configuration. The same pipeline code can run on a laptop, Kubernetes, Airflow, Kubeflow, or cloud services.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [Build vs Buy an ML Platform](https://mlopstalks.com/talks/build-vs-buy-an-ml-platform) (Diego Oppenheimer, Algorithmia, 57:20)
- [Building an ML Platform: Insights, Community, and Advocacy](https://mlopstalks.com/talks/building-an-ml-platform-insights-community-and-advocacy) (Stephen Batifol, Wolt, 45:49)
- [Scaling ML Capabilities in Large Organizations](https://mlopstalks.com/talks/scaling-ml-capabilities-in-large-organizations) (Bertjan Broeksema & Axel Goblet, BigData Republic, 1:02:47)
- [The Future of ML and Data Platforms](https://mlopstalks.com/talks/the-future-of-ml-and-data-platforms) (Michael Del Balso, Tecton, 55:17)
- [The Shipyard: Lessons Learned While Building an ML Platform](https://mlopstalks.com/talks/the-shipyard-lessons-learned-while-building-an-ml-platform) (Joseph Haaga, Interos, 40:00)