Andy McMahon defines MLOps as the practice of moving from one production model to an expanding, manageable portfolio of models.
2
Teams should establish their processes, culture, data foundations, and business goals before choosing tools or platforms.
3
LLM applications need their own evaluation and monitoring methods because ground truth is often unclear and metrics such as time to first token can mislead teams.
Summary
Andy McMahon argues that LLMOps should extend established software and MLOps practices rather than replace them. Teams still need requirements, development stages, testing, validation, deployment, monitoring, and automation. The differences come from the models, data, evaluation methods, and infrastructure used by generative AI systems. Andy recommends treating platforms as ecosystems of capabilities that teams can combine according to their needs. He also warns against choosing tools before understanding the workflow and the business problem. Traditional machine learning remains useful when it is cheaper, faster, more deterministic, or easier to audit than a generative model. For LLM applications, teams need to define meaningful product metrics, account for human evaluation, and avoid vanity measures such as time to first token. The conversation also covers migration in regulated banks, measuring time to value, leadership, psychological safety, and keeping technical work tied to customer and business outcomes.
LLMOps adds new pieces to the software development life cycle
Andy McMahon describes the life cycle as a familiar software development process with additional requirements for machine learning and generative AI. Teams still define requirements, work through development and test stages, deploy to production, validate changes, and monitor systems afterward. Infrastructure underpins the process, and automation should cover as much of it as possible. The harder questions concern what testing and monitoring mean when data affects the result and when model outputs are probabilistic. Andy says organizations often try to reinvent everything when a new technology arrives, then discover that much of their existing software and operational practice still applies.
MLOps begins when teams can repeat production delivery
Andy defines MLOps through the move from one production model to an expanding portfolio. Getting the first model into production is a major achievement, but it is only the beginning. Teams then need to handle multiple models, replacement models, models running in parallel, and champion-challenger arrangements. The challenge is to manage the growing estate in a repeatable way. Demetrios Brinkmann says he uses Andy's definition because it captures the shift from n equals one to n plus one models. Andy compares the process to proof by induction: a team must show that it can repeat the work rather than celebrate a single successful deployment.
Andy criticizes the idea that DevOps, MLOps, or AI infrastructure can be reduced to a collection of tools. He calls this silver-bullet thinking. In his view, a team with a strong process and capable people could accomplish the work with very basic technology. Data and the surrounding ecosystem come before tools in the order of concern. He encourages teams to push open source as far as possible before buying a product. That experience makes them better buyers because they learn which features matter and where their real gaps are. If provisioning cloud accounts is the obstacle, for example, buying a sophisticated retrieval application will not solve the problem.
Architecture should preserve workflows across interchangeable components
Demetrios connects Andy's approach to HashiCorp's idea of putting workflows before technology. Andy says a useful architecture lets teams replace one component with a competitor's product without changing the overall workflow. Interoperability, standard APIs, and clear contracts make this possible. Organizations can then build best-of-breed architectures, using different products for monitoring, scalable training, and data engineering. Andy adds that large banks still have legacy systems and bespoke components that cannot always be replaced. Migration decisions must include decommissioning costs, new infrastructure, maintenance, hiring or training, and the long-term target state. Sometimes the right decision is to leave a legacy system in place.
Large regulated organizations scale change horizontally
Andy explains that a bank cannot move at the same pace as a technology startup because stability, trust, governance, and customer data matter. Decades-old banking systems may remain because they work and are well maintained. Large migrations are handled by dividing the work into smaller pieces and running them in parallel. While one part of the bank moves first and gains value, other parts continue their own migration. Andy contrasts this horizontal scaling with startup work, where one person or team may complete a task end to end before moving on. In a large organization, several longer pieces of work progress at the same time and eventually produce a broad impact.
An AI platform should provide an ecosystem rather than separate silos
Andy thinks about AI and ML capabilities as an ecosystem or a shelf of tools that teams can use to experiment and build products. Traditional machine learning workflows remain important. Teams still need data access, exploratory analysis, development workflows, orchestration, and monitoring. The main difference is that LLM applications usually consume models provided as a service or commodity instead of training those models themselves. The work shifts toward vector databases, chunking and indexing, retrieval flows, and interactions with language models. Andy expects hybrid systems in which an LLM calls proprietary models, such as fraud detection models, and combines their outputs. Those models will need clear interfaces, validation, and organizational access.
The right level of abstraction depends on the user's needs
Andy describes several levels at which people may use AI. Some will consume copilots built into email, word processors, or coding environments. Others will use graphical tools such as Microsoft Copilot Studio to assemble applications without deep technical skills. AI and ML engineers will build bespoke systems when they need full control over the components and workflow. The same range applies to infrastructure. A team can use a hosted application, build its own front end around a model provider, manage its own vector database and chunking strategy, or run a model directly on its own hardware. Higher abstraction is easier but can cost more. Lower abstraction gives more control but requires more work and specialist skills.
Generative AI should be chosen for a real product outcome
Andy warns against using GenAI simply because it is available. A classification task may work better with logistic regression because it is cheaper, faster, more controlled, and easier to audit. A traditional model also produces a more stable result for the same inputs, while a third-party generative model can change over time. He gives recommender systems as an example where directly asking an LLM for recommendations may be too slow and expensive. A better design might use a traditional recommendation engine for the first set of results and an LLM to synthesize the response. Product managers and people with a product mindset help teams decide where generative AI adds value.
LLM evaluation needs business metrics and more than latency
Andy says evaluation and monitoring are harder for LLM systems because ground truth is often unclear. A chatbot may have no single correct answer, although specific retrieval tasks can use measures such as retrieval precision. Teams may need human evaluation, LLM-as-a-judge methods, or specialized guardrail models. They should also select metrics that reflect the product outcome. Time to first token may improve while users abandon the chatbot after a few lines because the answers are poor. Andy prefers measures such as whether a task was completed, how long users remain with the bot, and the sentiment of the interaction. He connects this to the Lean Startup idea of avoiding vanity metrics and asking what the system is actually meant to achieve.
"Fundamentally it's like you say, so getting one model in a production, the first model is always a big achievement for an organization, a team, a company. They're always super excited. But that's sort of day zero, day one, day two is all I need to go back and do it again."Andy McMahon10:34
Who should watch
You are designing an ML or AI platform and need a way to choose tools without losing sight of the workflow.
Your team is moving generative AI applications toward production and needs practical ways to evaluate quality, cost, and business value.
You work in a regulated organization where legacy systems, governance, and migration risk shape platform decisions.
