Companies should assess the business impact and cost of an LLM use case before building it.
2
Customer-facing LLM applications need risk assessment for hallucinations, privacy, bias, security, outages, lack of oversight, and misuse.
3
LLMOps keeps the core MLOps practices and adds tracking for prompts, model APIs, embeddings, retrieval, fine-tuning, and token costs.
Summary
Maria Vechtomova and Başak Tuğçe Eskili explain how teams can decide whether an LLM application makes sense and how to operate it responsibly. They describe common uses such as question-answering systems, chatbots, and agents, then compare foundation models, pretrained models, and models trained from scratch. Examples from Zalando, Booking.com, Pak'nSave, and DPD show both useful applications and failures caused by unedible recipes or a chatbot criticizing its own company. Their recommended process starts with impact assessment: define the business problem, identify users, estimate cost and time, and set measurable goals. Risk assessment follows, with attention to hallucination, privacy, bias, security, external API outages, human oversight, and misuse. They then present an LLM maturity assessment based on documentation, accessibility, reproducibility, code quality, monitoring, testing, resource use, and explainability, with additional checks for prompts, embeddings, retrieval, fine-tuning, and model APIs.
LLM applications are spreading through three common patterns
Başak Tuğçe Eskili describes question-answering systems, chatbots, and agents as the main application types they see in organizations. Chatbots are the most common. Agents take actions after a user request, such as sending emails or creating a web interface. She also describes three ways to use LLMs: call a foundation model endpoint with prompt engineering or retrieval augmentation, adapt a pretrained model with fine-tuning or retrieval, or train a model from scratch. The first two approaches are the most popular.
Retrieval and fine-tuning add domain-specific behavior
A retrieval-augmented system embeds a user prompt, searches a vector database for similar context, adds that context to the prompt, and sends it to an LLM. More advanced systems can rewrite, expand, or route the query before retrieval, then rank, summarize, or combine retrieved results afterward. Fine-tuning lets a team adapt a pretrained model to a particular task and bring in domain data. The speakers mention supervised and self-supervised fine-tuning, reinforcement learning, transfer learning, and parameter-efficient fine-tuning.
A simple business solution may be better than an LLM
Maria Vechtomova says an impact assessment should begin by identifying the business problem and the people who own it. Teams should ask whether an LLM is needed at all, since a simpler approach may meet the business goal. She warns that data scientists can start projects around interesting techniques without a business owner, which leaves the work disconnected from company objectives. Teams should estimate cost, delivery time, and expected impact before starting. If the cost exceeds the impact, the project does not make sense.
Customer-facing systems need explicit risk assessment
The speakers distinguish internal knowledge systems from customer-facing applications, where failures can reach the news and cause greater harm. Risks include hallucinations, privacy problems involving personally identifiable information, bias, data security breaches, dependence on external APIs, missing human oversight, and misuse. They give two examples. Pak'nSave's recipe tool produced recipes from inedible ingredients, while DPD's chatbot criticized DPD when asked about other delivery firms. These examples show why teams cannot assume that an LLM will stay within its intended use.
Maturity assessment should happen at the project level
Maria and Başak created a maturity assessment for MLOps and applied it across projects in a company with multiple brands and applications. Maria contrasts this with organization-level assessments from Google or Microsoft, which she says are less actionable for changing a specific project. Their assessment covers documentation, accessibility, reproducibility, code quality, monitoring, testing, resource use, and explainability. Documentation should include business goals, KPIs, architecture, model choice, and the reasons for selecting a model.
LLM systems need detailed lineage beyond standard model tracking
For a foundation model API, teams should record the endpoint and version, request and response structure, token usage, cost, latency, prompt, and response. Retrieval systems need records for embedding generation, storage, and retrieval. That includes the embedding model, computational and retrieval latency, document parsing, chunk size and strategy, metadata, update behavior, number of retrieved chunks, similarity algorithm, and metadata filtering. Prompt engineering needs a documented strategy for adding content to the user query. Fine-tuning needs records of code, infrastructure, artifacts, training data, retraining frequency, automation, and method.
Maria says LLM applications still need version control, CI/CD, orchestration, model registries, compute and serving, evaluation, monitoring, and feature stores. She names Airflow for orchestration and mentions SageMaker, Vertex AI, and MLflow for model registries. LLM systems add foundation model APIs, vector databases, prompt engineering interfaces, and sometimes different compute and serving requirements for fine-tuning. She expects vector databases to remain useful outside LLM applications, including recommendation systems.
Evaluation should start with measurable business outcomes
During the questions, the speakers return to business value. They suggest defining KPIs before choosing an LLM, such as improving a particular experience or increasing sales, then checking during impact assessment whether the system delivers that value. They describe a recipe example where OCR, a recommender system, and similarity matching could scan a recipe and add ingredients to a basket without using an LLM. Başak also says that in food retail they had a generative AI lab but no use case that generated serious value at that point.
"For any API call we can look up which endpoint and version was used, the structure of request and response, token usage, cost, latency, what prompt and response was generated."Maria Vechtomova15:45
Who should watch
You are deciding whether a proposed LLM feature has a real business owner, measurable value, and a cost that the organization can justify.
Your team operates a chatbot, retrieval system, or agent and needs a practical list of risks and records to track in production.
You already have MLOps practices and want to understand which prompt, embedding, retrieval, fine-tuning, and API details need to be added for LLM applications.
