# Culture and Architecture in MLOps

Jet Basrawi, Satalia | MLOps Coffee Sessions | Episode 29 | 53:41

Source: https://www.youtube.com/watch?v=uV676_YLP98
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/culture-and-architecture-in-mlops
Published: 2021-02-08
Tags: engineering-culture, platform-teams

## TL;DR
- Jet Basrawi argues that MLOps and DevOps share the same goal, high-performance software delivery, while machine learning adds challenges around data, access, and team boundaries.
- Jet says MLOps depends on leadership, trust, small batches, and grassroots communities of practice because technical practices fail when organizational culture blocks them.
- Jet argues that teams should simplify machine learning work, grow people through smaller disciplines and apprenticeships, and avoid adding accidental complexity through fashionable tools and patterns.

## Summary
Jet Basrawi frames MLOps as a cultural and organizational problem before it is a tooling problem. He sees MLOps and DevOps as versions of the same effort to deliver software with throughput and stability, while machine learning adds data access, annotation, versioning, and cross-team friction. Organizations often struggle because their past successes shaped cultures that do not fit iterative software delivery. Jet argues that leadership must create trust, support small batches, accept failure, and connect engineers to the outcomes of their work. He prefers grassroots communities of practice and apprenticeship models over large initiatives or demands for fully formed specialists. Tools and patterns can reduce unnecessary complexity, but they can also create new complexity when teams adopt them without understanding the problem. He is also skeptical of ritualized Agile practices. Scrum ceremonies can give unfamiliar teams a shared routine, but high-performing work depends on frequent conversation among people who understand the customer and each other.

## Key ideas
### MLOps and DevOps pursue the same delivery outcome
[06:00](https://www.youtube.com/watch?v=uV676_YLP98&t=360s)
Jet Basrawi says MLOps and DevOps are the same when viewed at a high level because both aim for high-performance software delivery, with throughput and stability. The purpose is to get machine learning models into use and deliver value quickly. He warns that teams can lose sight of those outcomes when they focus on artifacts, tools, and outputs. Machine learning does diverge from ordinary software delivery at a lower level because data introduces extra complications. Data access, data versioning, feature management, and ownership across teams can take much longer than the technical work itself.

### The hard part is often access to data and cooperation between teams
[09:08](https://www.youtube.com/watch?v=uV676_YLP98&t=548s)
Jet says data scientists create the value in machine learning work, yet they often do not care about infrastructure. The organization still has to provide the infrastructure and decide who will handle it. He describes data access as a frequent source of delay, with some teams taking months to obtain what they need. Feature management adds another problem after access is available. Jet agrees that much of the friction is organizational. Conflicts between teams and professions later get labelled as MLOps problems, even though the underlying issue is how people work together.

### Past success can make an organization resistant to iterative delivery
[12:31](https://www.youtube.com/watch?v=uV676_YLP98&t=751s)
Jet describes culture as the behaviours that led an organization to success in the past. A technology company may have developed habits that support software delivery, while Airbus developed habits for building airplanes. Those habits can make iterative software work difficult. In some organizations, teams must present the architecture of an entire system before receiving funding and resources. That requirement conflicts with learning through iteration because the team must claim to know what it will build before it has learned from the work. Jet says the people controlling money often have the strongest reasons to preserve the existing structure.

### Leadership should build trust and prepare for failure
[16:58](https://www.youtube.com/watch?v=uV676_YLP98&t=1018s)
Jet separates leadership from management. He describes management as handling business dysfunction, while leadership gives people an orientation and creates the conditions for good work. He points to high-trust, generative cultures as part of strong safety outcomes. His advice is to work in small batches instead of accepting a backlog of hundreds of items for a multi-year project. Teams should also assume that failure will happen and design around it with safeguards and a good incident-management process. Jet is careful that accepting failure does not mean ignoring it. Teams need a way to use failures to learn.

### Architecture decisions become dangerous when people cannot reverse them
[23:40](https://www.youtube.com/watch?v=uV676_YLP98&t=1420s)
Jet criticizes the idea that an architect can make all the right technical decisions in advance. He gives an example of a system component with a two-million-pound licence fee and a yearly cost of 250,000 pounds. The component was eventually disliked and only moved three CSV files, yet the organization resisted replacing it because it had been presented to the client as a central part of the architecture. Jet says organizational incentives can overpower technical judgement. People may keep a bad decision because changing it would make the earlier decision-makers look wrong. Psychological safety and the ability to revise decisions are therefore part of the architecture problem.

### Grassroots practice grows MLOps culture better than an ivory-tower initiative
[30:01](https://www.youtube.com/watch?v=uV676_YLP98&t=1801s)
Jet says the effective way to spread DevOps and MLOps practices is through grassroots work. He points to communities of practice and the State of DevOps reports, then describes sharing experience with nearby colleagues rather than launching a grand company-wide programme. He wants leaders to work with development teams instead of directing them from an ivory tower. Jet also argues for apprenticeship models because the work is complex and companies cannot solve staffing problems by hiring people who already have every required skill. Smaller disciplines can help people become effective sooner.

### Tools and patterns reduce complexity only when teams use them with judgement
[35:22](https://www.youtube.com/watch?v=uV676_YLP98&t=2122s)
Jet expects MLOps tools to remove non-essential complexity from machine learning work. That should let people focus on the essential problems and make it easier to bring people into the field through non-traditional routes. He warns that tools can also introduce accidental complexity. The same applies to design patterns. Jet recalls a project where a developer had applied so many patterns from the Gang of Four book that nobody could find the configuration. Patterns are useful when they fit the problem. Filling a project with patterns and tools without understanding their purpose can make the system harder to change.

### Ritualized Agile gives teams a shared routine, but conversation does the real work
[44:16](https://www.youtube.com/watch?v=uV676_YLP98&t=2656s)
Jet calls the standard weekly Agile routine of planning, stand-ups, and retrospectives a poor version of Agile, although it remains useful because most people know how to perform it. That shared routine helps a new group orient itself, but it does not make the group genuinely agile. He describes effective Agile work as a small group of people who know each other, stay focused on a customer problem, talk frequently, and deliver small amounts of work. Jira can become a substitute for communication. Jet says software development is a social activity, so cross-functional teams need direct conversations rather than relying on process records.

## Notable quotes
- "The real kung fu is in the culture of MLOps." (02:00)
- "They're both about high performance software delivery, about throughput and stability." (06:41)
- "DevOps is a business concern, not a technical concern." (18:18)
- "The architect metaphor is dead. I hate the architect metaphor now." (24:40)
- "Software development is a social activity you do with your friends." (47:24)

## Tools & references mentioned
- MLOps Community
- Demetrios Brinkmann
- DevOps
- Kubeflow
- Jess Humble
- David Ronchick
- Google
- Airbus
- Deloitte
- Accenture
- Toyota
- Edward Deming
- Conway's law
- Sam McNeill
- Workday
- Flavio
- Gang of Four
- State of DevOps reports
- Jira
- Agile Manifesto
- SAFE

## Who should watch
- You are introducing MLOps practices into a company whose processes were built around hardware, regulation, finance, or another older source of success.
- Your team keeps discussing tools and architectures while data access, ownership, handoffs, and leadership support remain unresolved.
- You want to grow MLOps capability without waiting to hire people who already combine deep software, data, infrastructure, and machine learning experience.

## Related talks

- [MLOps as Tool to Shape Team and Culture](https://mlopstalks.com/talks/mlops-as-tool-to-shape-team-and-culture) (Ciro Greco, Coveo, 43:02)
- [The Motivation for MLOps](https://mlopstalks.com/talks/the-motivation-for-mlops) (Steven Fines, CoreLogic, 56:42)
- [MLOps in Practice: Common Challenges and Lessons Learned](https://mlopstalks.com/talks/mlops-in-practice-common-challenges-and-lessons-learned) (Marouen Hizaoui & Mo Basirati, Machine Learning Reply, 1:03:01)
- [MLOps: Isn't That Just DevOps?](https://mlopstalks.com/talks/mlops-isnt-that-just-devops) (Ryan Dawson, Seldon, 1:06:32)
- [I Am Once Again Asking "What is MLOps?"](https://mlopstalks.com/talks/i-am-once-again-asking-what-is-mlops) (Oleksandr Stasyk, Synthesia, 1:06:22)