LLMOps changes the workflow because teams can start with a model in production and improve prompts and context before they collect data or train a model.
2
Many enterprise applications will still use traditional MLOps after a general-purpose model is used to create a smaller specialist model.
3
LLM systems need evaluation, tracing, and context debugging because prompts, vector retrieval, and chains of model calls can produce results that are hard to diagnose.
Summary
The panel compares the operating practices around traditional machine learning with those emerging around large language models. Willem Pienaar says LLM applications can begin with a model in production, while training and data collection become later steps. Chris Van Pelt adds that prompt and context iteration changes who can build these systems, although making a demo reliable remains difficult. Alex Ratner expects many enterprise use cases to start with a general model and end with a smaller specialist model running on familiar MLOps infrastructure. The panel discusses privacy, cost, open models, hosted APIs, and vector stores that connect models to private data. Aparna Dhinakaran describes a need to observe retrieved context and trace complex chains of calls. The panel agrees that evaluation is difficult because teams need representative test data and task-specific benchmarks. The closing view is that LLMOps will add practices and systems, but MLOps does not need an immediate rebrand.
LLM applications can start in production before teams build an offline training flow
Willem Pienaar says LLMOps is language-oriented and begins with how a builder frames a use case around a general model. The offline flow, including data collection and training, can become optional at the start. A team can put a model in front of users, observe traffic, and decide what to do next. This reverses the usual sequence in which teams first prepare data and train a model. Chris Van Pelt adds that the offline work may involve prompt iteration and context selection rather than training or fine-tuning. This lowers the barrier for people who did not previously work with traditional machine-learning stacks.
Prompt and context iteration broaden who can build machine-learning applications
Chris Van Pelt says improvement in LLM systems often comes from changing the prompt or the context supplied to it. That differs from the older workflow, where teams commonly trained a model to gain a measurable performance increase. He expects the shift to change team roles and says he has already seen job postings for LLM engineers. The panel does not treat fast experimentation as proof of production readiness. Van Pelt says it is easy to make a demo and much harder to make a system work consistently. The new workflow changes the work required, but it does not remove the engineering needed to operate the result.
Many enterprise systems will turn general models into smaller specialists
Alex Ratner separates open-ended agents and chat systems from predictive tasks such as classification and extraction. He expects many valuable enterprise applications to remain suitable for smaller, specialist models. A large foundation model can provide a starting point, or a warm start, and a smaller model can then be trained for a narrow problem. Ratner describes this as moving from a generalist to a specialist. He says the resulting artifact may look like a traditional machine-learning model, with serving infrastructure that resembles existing MLOps. Only applications that genuinely need a general-purpose model will require a substantially different serving setup.
The main training decision becomes how to supply data and context
Ratner says LLM application work may involve prompt engineering, examples in the prompt, fine-tuning, or external context. These approaches all depend on engineering the data and context given to the model. He expects teams to spend less time choosing hyperparameters, algorithms, and model architectures than they did in older machine-learning workflows. A specialized use case may need an adapter or fine-tuning, while another may work with a prompt and retrieved examples. The panel describes this as a range of customization options rather than a choice between a hosted API and building a model from scratch.
Enterprise teams choose among hosted, open, and private models based on data and cost
Chris Van Pelt says security and data teams worry about sending enterprise data to a third-party API. Some organizations may use an isolated Azure-native API, while others cannot send data to any third party and may need to build or operate more of the stack themselves. Alex Ratner says organizations can also start with an open model and specialize it for a narrow task. The panel discusses cost as an unresolved concern, especially when a large model must be managed and deployed. Ratner says owning a foundation model can make sense when many downstream teams use it, because shared work can support multiple projects.
Vector stores connect general models to private knowledge without retraining the whole model
Aparna Dhinakaran describes vector stores as a way to add a company's own documents and knowledge to an off-the-shelf model. The application supplements a user query with context retrieved from a private knowledge base. This can support use cases such as answering questions about documents without pre-training or fine-tuning the entire model whenever new material is added. The panel presents this as a simpler option than building or training a private LLM when a public model is otherwise suitable. If the retrieved context does not produce good responses, a team may move to fine-tuning or eventually to an open model it can operate itself.
LLM observability must inspect retrieved context and long chains of calls
Dhinakaran says LLM observability needs to show whether the right context was retrieved from a vector store. Teams need to inspect whether the selected documents were relevant, rather than assuming that the most similar documents were the right ones. LLM applications can also contain chains of calls to different models and services. Chris Van Pelt says a failure in any part of such a chain can produce a bad final result. These systems therefore need tracing and diagnosis across the full sequence. The panel connects this work to an evaluation stack that can measure whether a system remains useful after it leaves the demo stage.
Evaluation requires representative test data and task-specific benchmarks
Alex Ratner says evaluation is difficult because teams need test data that represents what the system will do in production. LLM applications may not have a single simple accuracy score. Teams need custom benchmarks and must know where the evaluation data came from, including whether it may have appeared in the training data of a large model. Ratner says impressive demos often look different after rigorous evaluation. The panel treats evaluation as an area that is still developing, especially for systems with retrieval, multiple tools, and several model calls. Measuring quality is part of operating the system, not a check that can be skipped after the demo works.
MLOps remains the name for now, while LLMOps adds practices around new system types
The panel rejects the idea that MLOps must immediately be renamed. Chris Van Pelt says measuring systems and keeping an auditable record remain relevant. Aparna Dhinakaran expects LLMs to replace some traditional models for tasks such as sentiment analysis and classification, while traditional machine learning will remain common because LLMs can be expensive and unsuitable for some applications. Willem Pienaar says MLOps is easier to say and agrees that the field is still developing. Alex Ratner suggests that many foundation-model projects will eventually reduce to conventional machine learning and MLOps, while more general systems will need additional operational practices.
"It's very easy in the LLMOps world to try things really quickly, get a Twitter demo up, post it on Twitter, but just as in MLOps, very, very hard to get something to work robustly, work consistently."Chris Van Pelt07:05
Who should watch
You are deciding whether an existing MLOps platform can support applications built with large language models.
Your team needs to choose between prompt engineering, retrieval, fine-tuning, an open model, or a hosted API.
You are building an evaluation and observability process for systems that retrieve documents or call several models and services.
