# Who's MLOps for Anyway?

Jonathan Rioux, EPAM Systems | MLOps Podcast | Episode 261 | 1:10:15
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=HAIpMAZB_F4
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/whos-mlops-for-anyway
Published: 2024-09-17
Tags: build-vs-buy, platform-teams, rag, workflows

## TL;DR
- MLOps can make a bad idea reach production faster and more reliably, but it cannot make the idea valuable.
- Organizations should treat AI centers of excellence as support teams that help business functions do the right thing, rather than as central governing authorities.
- Teams should start with manual processes, automate the stable parts, and only then add LLMs or agents where the risks and failure modes are acceptable.

## Summary
Jonathan Rioux argues that MLOps is a set of principles, processes, and tools for moving an idea into production and maintaining it. It does not create good ideas or guarantee business value. He describes how AI hype pushes organizations toward technology-first use cases, centralized centers of excellence, and proofs of concept that later face awkward ROI demands. His preferred organizational model is a hub and spoke arrangement, where a central group supports business teams with platforms, conventions, and expertise while leaving them room to innovate. The conversation also covers change management, the difficulty of adopting tools such as version control and CI/CD, and the operational cost of bespoke infrastructure. For high-risk support applications, Jonathan recommends deterministic retrieval and classification instead of putting an LLM in charge of answers. He extends the same reasoning to agents: automate the underlying process first, then add agentic behavior only when the workflow, controls, and failure handling are ready.

## Key ideas
### MLOps improves delivery without creating business value
[02:03](https://www.youtube.com/watch?v=HAIpMAZB_F4&t=123s)
Jonathan defines MLOps as principles, processes, and tooling that help teams move an idea into production and maintain it there. The definition says nothing about whether the original idea is worthwhile. He gives a blunt example: a terrible idea implemented with MLOps will produce a terrible product faster and more reliably. Demetrios agrees that strong MLOps practices do not guarantee a successful ML product. Jonathan objects to organizations calling themselves "MLOps compliant" while a central shared service decides which projects are good enough. He thinks the original goal was to put business-facing people together with technology and ML practitioners so that ML could be embedded where it made sense.

### Technology-first AI programs lose sight of the business problem
[07:19](https://www.youtube.com/watch?v=HAIpMAZB_F4&t=439s)
Jonathan connects the current generative AI cycle with earlier deep learning hype. Organizations are starting with a new technology and asking where they can apply it, even though they understand their own business far better than they understand the technology. He recommends starting from known business problems and reviewing the portfolio as a whole. In his example, EPAM gets more value from small improvements to cumbersome, tedious, repetitive processes than from a grand innovation program. He says centralizing generative AI work can make this worse because data scientists already working inside products may be given another role in an AI center of excellence. The resulting disruption comes from the organizational response as much as from the technology.

### A central AI group should support teams instead of controlling intake
[18:54](https://www.youtube.com/watch?v=HAIpMAZB_F4&t=1134s)
Jonathan rejects both extremes: a central team that controls all data, ML, and LLM work, and fully independent business functions that each manage the entire stack. Centralization can leave one group unable to understand every business function deeply. Full decentralization makes governance, risk control, and the spread of good practices difficult. His recommendation in 2024 is a hub and spoke model in which the center of excellence is a support function. Its job is to provide tools, conventions, and help so that teams can do the right thing more easily. He compares the role to a scrum master who facilitates work. As LLMs become easier to understand, work may decentralize, until the next technology gap creates another centralization cycle.

### Platforms should support common work while allowing controlled exceptions
[22:04](https://www.youtube.com/watch?v=HAIpMAZB_F4&t=1324s)
Demetrios uses Uber's Michelangelo as an example of a platform that supports most use cases while allowing advanced teams to do their own work and bring it back into the common system. Jonathan likes the example, while warning that most organizations cannot copy Uber's bespoke architecture or its investment level. He says teams should begin with the jobs to be done: which processes people should follow and where the source of truth lives. People follow processes, and platforms enable them. A platform also needs a clear way for teams to go off script and reconnect later. Common activities such as hosting a model should be simple, with an understood service level and a realistic decision about whether to build or buy.

### Infrastructure choices can consume the team that was meant to build models
[24:49](https://www.youtube.com/watch?v=HAIpMAZB_F4&t=1489s)
Jonathan describes a previous organization that deployed models on EKS. The setup was intellectually rewarding, but a scaling problem mobilized the entire team. A month later, there was no new model refresh or other new work. He uses this experience to question the assumption that building infrastructure in-house is automatically cheaper or better. A data science team that lacks cloud competence may enjoy a self-built system while everything works, then struggle when an abstraction fails. His advice is to make the supported path easy and friendly. Teams should have conventions they can connect to when they need to depart from the standard process, rather than having to operate an entirely separate platform.

### Adoption takes longer than designing an MLOps process
[30:07](https://www.youtube.com/watch?v=HAIpMAZB_F4&t=1807s)
Jonathan says that implementing a stack, suggesting a process map, and comparing the current state with the desired state can be done fairly quickly. Getting people to adopt the tools takes much longer. He recalls interns struggling with version control, CI/CD, Makefiles, and automation even though those tools felt obvious to him and his team. Jonathan admits that it took him time to learn version control and CI/CD properly, yet he still tends to expect others to adopt them quickly. Change spreads from company decisions to teams and then to individual developers, and it can take months or years. People often need to feel enough frustration with the current process before they are ready to change it.

### A proof of concept should validate a small technical question before a business case
[35:27](https://www.youtube.com/watch?v=HAIpMAZB_F4&t=2127s)
Jonathan explains that many enterprises choose familiar generative AI experiments, such as an HR chatbot, to learn the technology. They may spend money building and maintaining the system, then ask for its ROI even though learning was the original reason for selecting it. An HR chatbot can be useful for gaining experience, but it may not reduce hiring or free people for revenue-generating work. He recommends defining the purpose at the start and using a scorecard. A proof of concept should often boil the work down to one, or at most three, questions. For document extraction, the question may be whether the model works on the organization's own documents. That result can inform a business case without building a full product first.

### High-risk chatbots can use deterministic retrieval and LLMs for content work
[50:19](https://www.youtube.com/watch?v=HAIpMAZB_F4&t=3019s)
For content where the tolerance for hallucination is zero, Jonathan describes a design that does not use an LLM to generate the answer. A classifier identifies the relevant article, while structured customer and policy information supplies the surrounding context. An LLM can help create and curate the content, generate question-and-answer pairs, and produce embeddings for indexing. The user-facing response can then present the applicable policy article with limited deterministic wording and escalate to a human. Jonathan gives insurance and medication interactions as examples where incorrect answers can create legal or safety problems. He calls the pattern a reverse RAG during the conversation, then explains that its value comes from using generative AI for the labor-intensive content work while keeping the final answer controlled.

### Agents should come after the underlying workflow is automated and understood
[1:02:04](https://www.youtube.com/watch?v=HAIpMAZB_F4&t=3724s)
Jonathan says function calling only works when the functions and underlying processes already exist. Closing a bank account may involve a decision tree, several applications, multiple screens, and a letter. The first step should be automating that procedure, then growing the process before giving an LLM more control. He recommends designing for failure because an LLM may make the wrong choice and humans also make mistakes. At every decision point, teams should ask whether the use case still works if the system is right 70 percent of the time and wrong 30 percent of the time. If zero errors are required, the use case is probably unsuitable for that model. This approach adds a manual, automated stage before an agentic one.

## Notable quotes
- Jonathan Rioux: "A terrible idea implemented using MLOps principle is going to yield a terrible product but faster and more reliably." (03:07)
- Jonathan Rioux: "The best model that we can have is some sort of Hub and spoke where you're not seeing the center of excellence as a governing function but more of a support function." (20:09)
- Jonathan Rioux: "One of the thing that we're implementing on a lot of the POCs that we do is we start with a scorecard." (44:57)
- Jonathan Rioux: "If there's nothing to call then everything fall like a house of cards." (1:03:04)
- Jonathan Rioux: "Let's assume that's 70% of the time it's going to do the right thing but 30% of the time it's going to do something wrong, is this making or breaking your use case?" (1:08:53)

## Tools & references mentioned
- MLOps Community
- EPAM Systems
- MLOps
- DevOps
- generative AI
- large language models
- Uber
- Michelangelo
- EKS
- MLOps consulting
- HR chatbot
- RAG
- reverse RAG
- LLM agents
- function calling
- RPA
- Air Canada
- Slack AI
- SAS Decision Builder
- Microsoft Fabric
- Luigi Patruno

## Who should watch
- You are deciding whether an AI center of excellence should own every use case or support teams that understand their own business processes.
- Your team is building generative AI proofs of concept and needs a way to separate technical learning from a later ROI claim.
- You are considering an LLM chatbot or agent for a regulated workflow where incorrect answers or uncontrolled actions would be costly.

## Editor's note

Jonathan Rioux describes how an EKS scaling problem consumed the team that was meant to build models. ZenML lets teams write ML workflows as Python pipelines and choose the orchestrator and other infrastructure by configuration, so the same pipeline code can run on a laptop, Kubernetes, Airflow, Kubeflow, or a cloud provider's services.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [Design and Development Principles for LLMOps](https://mlopstalks.com/talks/design-and-development-principles-for-llmops) (Andy McMahon, Barclays Bank, 1:10:18)
- [MLOps at the Crossroads](https://mlopstalks.com/talks/mlops-at-the-crossroads) (Patrick Barker, Kentauros AI & Farhood Etaati, AIMedic, 49:02)
- [I Am Once Again Asking "What is MLOps?"](https://mlopstalks.com/talks/i-am-once-again-asking-what-is-mlops) (Oleksandr Stasyk, Synthesia, 1:06:22)
- [Founding, Funding, and the Future of MLOps](https://mlopstalks.com/talks/founding-funding-and-the-future-of-mlops) (Mihail Eric, Storia AI, 57:31)
- [MLOps Critiques](https://mlopstalks.com/talks/mlops-critiques) (Matthijs Brouns, Xccelerated.io, 49:44)