Watch on YouTubeMost organizations are still stuck between GenAI proofs of concept and production deployment, especially at enterprise scale.
Enterprise GenAI requires different data, tooling, evaluation, cost models, and team structures from classical MLOps.
Organizations should reuse their existing MLOps and ML engineering practices while adding the new components needed for GenAI applications.
Andy McMahon argues that deploying LLM and GenAI systems at enterprise scale is difficult because organizations are still moving from experimentation into development and production. GenAI changes the problem framing, the data layer, the tools, the evaluation process, and the economics. Leaders need to assess foundation models by cost per query, latency, throughput, and whether they solve a business problem, rather than by leaderboard position alone. Enterprise teams also need vector databases, prompt hubs, application databases, human evaluation, LLM evaluation, and guard rails. The organizational response includes centers of excellence, adaptable architecture teams, and closer collaboration between software developers, data scientists, and ML engineers. Andy advises most organizations not to pre-train their own models and to use scalable fine-tuning and optimization techniques when needed. His broader point is practical: companies can reuse much of their existing MLOps foundation, then add the new GenAI-specific pieces and processes.
Andy says deployment of LLMs and GenAI is hard, even though most organizations are already using generative AI in some form. In his four-stage machine learning lifecycle, teams move through Discover, Play, Develop, and Deploy. Many organizations are currently stuck at Develop. The problem is especially difficult in large companies, where infrastructure, budgets, approvals, and operating practices take time to change.
Classical MLOps and LLMOps differ in several practical ways. Teams now work with prompts and context rather than mainly tabular data and features. They use different pipeline and orchestration tools, and they need different metrics. The business question also changes from asking whether a problem suits classification, regression, or unsupervised learning to asking whether a generative approach can solve it.
Andy is less interested in which model leads a Hugging Face leaderboard than in whether it solves the required task at an acceptable cost. He suggests considering cost per query, cost per user response per day, speed, latency, throughput, and the expected business benefit. Enterprise model selection is therefore tied to a clear value equation and a budget.
The new stack includes familiar areas such as orchestration, monitoring, logging, and caching, alongside newer components such as embedding models, vector databases, and tools for coordinating LLM applications. Andy says large organizations struggle because they often have long cycles for approving budgets, infrastructure, and tools. They need to become more adaptable as the stack changes.
Pre-training a model requires a major investment, and fine-tuning can also be expensive. Storage grows because models and the data used by large retrieval systems are substantial. Latency and infrastructure costs also matter. Andy recommends that most organizations avoid pre-training their own model, use scalable frameworks for fine-tuning, and apply established techniques such as quantization and caching.
GenAI monitoring requires more than standard system metrics. Andy describes a workflow that combines objective ground truth, subject matter expertise, human evaluation, and LLM evaluation. Guard rails add another layer. A tool such as NeMo Guardrails can be configured easily, but applying consistent rules across very large teams is harder.
At NatWest Group, Andy describes a population of 500 data scientists and engineers who need to work from consistent practices. He recommends centers of excellence that spread learning across use cases and architecture teams that connect platform-level decisions with individual applications. Team structures are also changing, with software developers working more closely with ML engineers and data scientists to build interfaces, database layers, and GenAI applications.
Andy compares the current GenAI moment with the earlier rise of data science, when many organizations struggled to put solutions into production. He expects MLOps practices to mature GenAI development in a similar way. Organizations do not need to rebuild everything. Their ML engineering and MLOps foundations can remain in place while they add new components, processes, and operating models.
"I'm not too interested in what's topping the Hugging Face leaderboard. I'm more interested in things like what's the cost per query."02:30