# Design and Development Principles for LLMOps

Andy McMahon, Barclays Bank | MLOps Podcast | Episode 254 | 1:10:18
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=LHEKLqpI2VY
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/design-and-development-principles-for-llmops
Published: 2024-08-20
Tags: build-vs-buy, evals, observability, platform-teams

## TL;DR
- Andy McMahon defines MLOps as the practice of moving from one production model to an expanding, manageable portfolio of models.
- Teams should establish their processes, culture, data foundations, and business goals before choosing tools or platforms.
- LLM applications need their own evaluation and monitoring methods because ground truth is often unclear and metrics such as time to first token can mislead teams.

## Summary
Andy McMahon argues that LLMOps should extend established software and MLOps practices rather than replace them. Teams still need requirements, development stages, testing, validation, deployment, monitoring, and automation. The differences come from the models, data, evaluation methods, and infrastructure used by generative AI systems. Andy recommends treating platforms as ecosystems of capabilities that teams can combine according to their needs. He also warns against choosing tools before understanding the workflow and the business problem. Traditional machine learning remains useful when it is cheaper, faster, more deterministic, or easier to audit than a generative model. For LLM applications, teams need to define meaningful product metrics, account for human evaluation, and avoid vanity measures such as time to first token. The conversation also covers migration in regulated banks, measuring time to value, leadership, psychological safety, and keeping technical work tied to customer and business outcomes.

## Key ideas
### LLMOps adds new pieces to the software development life cycle
[07:27](https://www.youtube.com/watch?v=LHEKLqpI2VY&t=447s)
Andy McMahon describes the life cycle as a familiar software development process with additional requirements for machine learning and generative AI. Teams still define requirements, work through development and test stages, deploy to production, validate changes, and monitor systems afterward. Infrastructure underpins the process, and automation should cover as much of it as possible. The harder questions concern what testing and monitoring mean when data affects the result and when model outputs are probabilistic. Andy says organizations often try to reinvent everything when a new technology arrives, then discover that much of their existing software and operational practice still applies.

### MLOps begins when teams can repeat production delivery
[10:07](https://www.youtube.com/watch?v=LHEKLqpI2VY&t=607s)
Andy defines MLOps through the move from one production model to an expanding portfolio. Getting the first model into production is a major achievement, but it is only the beginning. Teams then need to handle multiple models, replacement models, models running in parallel, and champion-challenger arrangements. The challenge is to manage the growing estate in a repeatable way. Demetrios Brinkmann says he uses Andy's definition because it captures the shift from n equals one to n plus one models. Andy compares the process to proof by induction: a team must show that it can repeat the work rather than celebrate a single successful deployment.

### Process and culture come before tool selection
[12:11](https://www.youtube.com/watch?v=LHEKLqpI2VY&t=731s)
Andy criticizes the idea that DevOps, MLOps, or AI infrastructure can be reduced to a collection of tools. He calls this silver-bullet thinking. In his view, a team with a strong process and capable people could accomplish the work with very basic technology. Data and the surrounding ecosystem come before tools in the order of concern. He encourages teams to push open source as far as possible before buying a product. That experience makes them better buyers because they learn which features matter and where their real gaps are. If provisioning cloud accounts is the obstacle, for example, buying a sophisticated retrieval application will not solve the problem.

### Architecture should preserve workflows across interchangeable components
[17:15](https://www.youtube.com/watch?v=LHEKLqpI2VY&t=1035s)
Demetrios connects Andy's approach to HashiCorp's idea of putting workflows before technology. Andy says a useful architecture lets teams replace one component with a competitor's product without changing the overall workflow. Interoperability, standard APIs, and clear contracts make this possible. Organizations can then build best-of-breed architectures, using different products for monitoring, scalable training, and data engineering. Andy adds that large banks still have legacy systems and bespoke components that cannot always be replaced. Migration decisions must include decommissioning costs, new infrastructure, maintenance, hiring or training, and the long-term target state. Sometimes the right decision is to leave a legacy system in place.

### Large regulated organizations scale change horizontally
[26:26](https://www.youtube.com/watch?v=LHEKLqpI2VY&t=1586s)
Andy explains that a bank cannot move at the same pace as a technology startup because stability, trust, governance, and customer data matter. Decades-old banking systems may remain because they work and are well maintained. Large migrations are handled by dividing the work into smaller pieces and running them in parallel. While one part of the bank moves first and gains value, other parts continue their own migration. Andy contrasts this horizontal scaling with startup work, where one person or team may complete a task end to end before moving on. In a large organization, several longer pieces of work progress at the same time and eventually produce a broad impact.

### An AI platform should provide an ecosystem rather than separate silos
[31:12](https://www.youtube.com/watch?v=LHEKLqpI2VY&t=1872s)
Andy thinks about AI and ML capabilities as an ecosystem or a shelf of tools that teams can use to experiment and build products. Traditional machine learning workflows remain important. Teams still need data access, exploratory analysis, development workflows, orchestration, and monitoring. The main difference is that LLM applications usually consume models provided as a service or commodity instead of training those models themselves. The work shifts toward vector databases, chunking and indexing, retrieval flows, and interactions with language models. Andy expects hybrid systems in which an LLM calls proprietary models, such as fraud detection models, and combines their outputs. Those models will need clear interfaces, validation, and organizational access.

### The right level of abstraction depends on the user's needs
[38:15](https://www.youtube.com/watch?v=LHEKLqpI2VY&t=2295s)
Andy describes several levels at which people may use AI. Some will consume copilots built into email, word processors, or coding environments. Others will use graphical tools such as Microsoft Copilot Studio to assemble applications without deep technical skills. AI and ML engineers will build bespoke systems when they need full control over the components and workflow. The same range applies to infrastructure. A team can use a hosted application, build its own front end around a model provider, manage its own vector database and chunking strategy, or run a model directly on its own hardware. Higher abstraction is easier but can cost more. Lower abstraction gives more control but requires more work and specialist skills.

### Generative AI should be chosen for a real product outcome
[40:25](https://www.youtube.com/watch?v=LHEKLqpI2VY&t=2425s)
Andy warns against using GenAI simply because it is available. A classification task may work better with logistic regression because it is cheaper, faster, more controlled, and easier to audit. A traditional model also produces a more stable result for the same inputs, while a third-party generative model can change over time. He gives recommender systems as an example where directly asking an LLM for recommendations may be too slow and expensive. A better design might use a traditional recommendation engine for the first set of results and an LLM to synthesize the response. Product managers and people with a product mindset help teams decide where generative AI adds value.

### LLM evaluation needs business metrics and more than latency
[48:07](https://www.youtube.com/watch?v=LHEKLqpI2VY&t=2887s)
Andy says evaluation and monitoring are harder for LLM systems because ground truth is often unclear. A chatbot may have no single correct answer, although specific retrieval tasks can use measures such as retrieval precision. Teams may need human evaluation, LLM-as-a-judge methods, or specialized guardrail models. They should also select metrics that reflect the product outcome. Time to first token may improve while users abandon the chatbot after a few lines because the answers are poor. Andy prefers measures such as whether a task was completed, how long users remain with the bot, and the sentiment of the interaction. He connects this to the Lean Startup idea of avoiding vanity metrics and asking what the system is actually meant to achieve.

## Notable quotes
- Andy McMahon: "Fundamentally it's like you say, so getting one model in a production, the first model is always a big achievement for an organization, a team, a company. They're always super excited. But that's sort of day zero, day one, day two is all I need to go back and do it again." (10:34)
- Andy McMahon: "The key thing is building the right stuff to put on that shelf for them to use or enabling it." (32:11)
- Andy McMahon: "We shouldn't just do GenAI to do GenAI." (40:25)
- Andy McMahon: "I don't care that like the first token came really fast, but it was the rest of the tokens were absolute garbage." (51:31)
- Andy McMahon: "My job is to make you look better. My job is to make you do better stuff." (1:03:18)

## Tools & references mentioned
- Barclays Bank
- NatWest Group
- Machine Learning Engineering with Python
- Oxford
- University of Warsaw
- OpenAI
- DeepMind
- MLOps
- LLMOps
- DevOps
- HashiCorp
- Excel
- open source
- AWS
- GCP
- GenAI
- SQL
- Microsoft Copilot Studio
- ChatGPT
- Perplexity
- Microsoft 365 Copilot
- llama.cpp
- vector databases
- RAG
- LangChain
- Llama
- Lean Startup
- Eric Ries
- DORA metrics
- Toyota Five Whys
- logistic regression
- Llama Guard

## Who should watch
- You are designing an ML or AI platform and need a way to choose tools without losing sight of the workflow.
- Your team is moving generative AI applications toward production and needs practical ways to evaluate quality, cost, and business value.
- You work in a regulated organization where legacy systems, governance, and migration risk shape platform decisions.

## Editor's note

Andy McMahon argues that teams should establish their processes, culture, data foundations, and business goals before choosing tools or platforms. ZenML lets teams express workflows as Python pipelines and run the same code on different configured stacks, from a laptop to Kubernetes or cloud services. This keeps infrastructure choices separate from the workflow teams are trying to build.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [MLOps vs LLMOps](https://mlopstalks.com/talks/mlops-vs-llmops) (Richa Sachdev, JPMorgan Chase & Willem Pienaar, Feast & Chris Van Pelt, Weights & Biases & Aparna Dhinakaran, Arize AI & Alex Ratner, Snorkel, 34:57)
- [Making Sense of LLMOps](https://mlopstalks.com/talks/making-sense-of-llmops) (Maria Vechtomova, Ahold Delhaize & Başak Tuğçe Eskili, Booking.com, 25:35)
- [Who's MLOps for Anyway?](https://mlopstalks.com/talks/whos-mlops-for-anyway) (Jonathan Rioux, EPAM Systems, 1:10:15)
- [MLOps at the Crossroads](https://mlopstalks.com/talks/mlops-at-the-crossroads) (Patrick Barker, Kentauros AI & Farhood Etaati, AIMedic, 49:02)
- [Enterprises Using MLOps, the Changing LLM Landscape, MLOps Pipelines](https://mlopstalks.com/talks/enterprises-using-mlops-the-changing-llm-landscape-mlops-pipelines) (Chris Van Pelt, Weights & Biases, 47:51)