# Beyond the Hype: Monitoring LLMs in Production Claire Longo, Arize | LLMs in Production 2023 | 16:47 Source: https://www.youtube.com/watch?v=ykQ9XIzUkqA Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks. Page: https://mlopstalks.com/talks/beyond-the-hype-monitoring-llms-in-production Published: 2023-07-05 Tags: embeddings, monitoring, observability, rag ## TL;DR - Monitoring an LLM system requires logging queries, responses, retrieved context data, embeddings, and human feedback. - Embedding spaces can reveal gaps between user queries and the context stored in a vector database. - Euclidean distance between embedding clusters can be used to monitor whether queries and context have become significantly different. ## Summary Claire Longo explains how monitoring an LLM system differs from monitoring a conventional tabular machine learning model. An LLM application may include a user query, retrieved context from a vector store, a prompt template, the model response, and user feedback. Each part creates data that can help diagnose failures. Longo recommends collecting queries, responses, context data, embeddings, and feedback. She focuses on embeddings because they preserve patterns in text as vectors of numbers. By visualizing query and context embeddings in latent space, teams can find cases where users ask about subjects that are poorly represented in the context store. Distance metrics can then turn those differences into something that can be monitored over time. She also discusses hallucinations, evaluation difficulties, and the trade-offs between simple API prototypes, prompt engineering, custom vector databases, fine-tuning, and serving a foundation model on private infrastructure. ## Key ideas ### LLM applications have production differences that are worth monitoring directly [01:39](https://www.youtube.com/watch?v=ykQ9XIzUkqA&t=99s) Claire Longo has worked on deploying conventional machine learning systems and sees LLM applications as having different production concerns. She prefers the broader MLOps term, but says LLM systems introduce nuances that deserve separate attention. Her talk applies lessons from monitoring machine learning models in production to systems built around LLMs. The main focus is finding what data flows through an LLM application and where that data can reveal failures. ### The technology choice depends on data privacy and business value [03:21](https://www.youtube.com/watch?v=ykQ9XIzUkqA&t=201s) Longo says teams should first ask whether the application requires proprietary data and whether generative AI adds enough value to justify its complexity. Proprietary data may push a team toward building in-house instead of sending data through an open API. The value question means comparing the proposed system with a simpler rules-based approach and considering both the cost of building it and the expected business return. ### LLM deployment options increase in complexity and control [04:55](https://www.youtube.com/watch?v=ykQ9XIzUkqA&t=295s) Longo describes a progression from a single endpoint prototype using an API, through prompt engineering, a custom vector database, fine-tuning, and serving a foundation model on private infrastructure. A basic API is quick to test but is not personalized to a business use case. Prompt templates add software and tuning. A vector database can supply documents or other context to improve answers. Fine-tuning costs more because the models contain many weights. Private serving gives the team more control and keeps data inside its infrastructure, but it is the most complex option. ### A typical LLM system combines queries, retrieved context, prompts, and feedback [07:57](https://www.youtube.com/watch?v=ykQ9XIzUkqA&t=477s) In Longo's example, a user query triggers a lookup in a vector store. The retrieved context, such as user documentation, is combined with the query through a prompt template. The prompt is sent to the LLM, which returns a response. User feedback can be collected after the response and used to create a feedback loop for improving the system. This architecture creates several places where monitoring data can be collected. ### Embeddings turn text into data that can be measured [09:05](https://www.youtube.com/watch?v=ykQ9XIzUkqA&t=545s) An embedding is a mathematical representation of text as a vector of numbers. Longo says embeddings preserve patterns in text that can be used by mathematical models. In an LLM application, embeddings may come from user queries, the context stored in a vector database, and model responses. Collecting these vectors gives a team data for studying how the different parts of the system relate to one another. ### Hallucinations and evaluation are harder when answers are not simply right or wrong [10:36](https://www.youtube.com/watch?v=ykQ9XIzUkqA&t=636s) Longo identifies hallucinations as a failure that can occur when the context data is poor or the model has not been properly fine-tuned. The model may respond confidently while making information up. She also says LLM evaluation is harder than evaluation for tabular machine learning because answers are not always simply correct or incorrect. Relevance can help, but it does not fully capture quality. User feedback can provide a useful proxy when the system has a meaningful business function. ### LLM monitoring keeps the usual audit principles but changes the data [12:08](https://www.youtube.com/watch?v=ykQ9XIzUkqA&t=728s) For a conventional recommender system, teams can log features, predictions, and truth labels. An LLM system instead requires logging queries, responses, retrieved context, and human feedback. Longo says the data is more complex, while the basic monitoring principles remain similar. The first practical step is to generate and save embeddings from the production system, either by extracting them from a neural network or by generating them through an API such as OpenAI or Hugging Face. ### Embedding distance can expose gaps between user demand and stored context [15:06](https://www.youtube.com/watch?v=ykQ9XIzUkqA&t=906s) Longo recommends visualizing embeddings in latent space and measuring distances between clusters. If users begin asking about information that is poorly represented in the context vector store, the gap between query embeddings and context embeddings can appear in the visualization. Teams can then monitor a distance metric, such as Euclidean distance. A large difference between the clusters can indicate that queries and available context have diverged. ## Notable quotes - "I do think there are some nuances with productionalizing the generative AI that really differentiate it enough from this old school kind of tabular ML." (02:01) - "The data that we want to log are no longer features and predictions. We're looking at queries and responses." (13:08) - "The embeddings are just very long vectors of numbers. They're super meaningful, and I'll show you how we can get meaning out of them." (13:47) - "You can visualize this. You could look at your embeddings for your queries, you can look at your embeddings for your context store, you can measure different distances between that." (15:23) ## Tools & references mentioned - Arize - Arize Phoenix - OpenAI - Hugging Face ## Who should watch - You are building a retrieval-augmented LLM application and need to decide what to log from the query, retrieval, prompt, and response path. - Your team monitors tabular models today and needs a concrete way to think about embeddings, latent space, and drift in unstructured data. - You need to choose between an API prototype, retrieval context, fine-tuning, or private model serving and want the monitoring implications of each option. ## Related talks - [Current State of LLMs in Production](https://mlopstalks.com/talks/current-state-of-llms-in-production) (Apurva Misra, Truckstop, 11:46) - [LLMs in Production Conference - Part II](https://mlopstalks.com/talks/llms-in-production-conference-part-ii) (Matei Zaharia, Databricks and Stanford & Chip Huyen, 37:21) - [Embeddings and Retrieval for LLMs: Techniques and Challenges](https://mlopstalks.com/talks/embeddings-and-retrieval-for-llms-techniques-and-challenges) (Anton Troynikov, Chroma, 35:19) - [Making Sense of LLMOps](https://mlopstalks.com/talks/making-sense-of-llmops) (Maria Vechtomova, Ahold Delhaize & Başak Tuğçe Eskili, Booking.com, 25:35) - [Enterprises Using MLOps, the Changing LLM Landscape, MLOps Pipelines](https://mlopstalks.com/talks/enterprises-using-mlops-the-changing-llm-landscape-mlops-pipelines) (Chris Van Pelt, Weights & Biases, 47:51)