# Building LLM Applications for Production

Chip Huyen, Claypot AI | LLMs in Production 2023 | 35:23

Source: https://www.youtube.com/watch?v=spamOhG7BOA
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/building-llm-applications-for-production
Published: 2023-06-29
Tags: data-quality, edge, privacy, reliability

## TL;DR
- LLM applications can produce different outputs for the same or slightly changed inputs, which makes downstream parsing and application behavior difficult to control.
- Hallucinations, privacy, context length, model and data drift, edge deployment, language coverage, and interface design remain open production problems.
- Companies building with generative AI need a clear data strategy because training data can become outdated, scarce, or contaminated by AI-generated content.

## Summary
Chip Huyen discusses the production problems that remain after an LLM can generate a convincing response. Small input changes can cause large output changes, and temperature zero does not make applications fully deterministic. This creates trouble when software needs to parse a reliable schema. She covers hallucinations in factual, legal, contract-processing, and text-to-SQL tasks, along with privacy risks when models handle sensitive data. Other challenges include context length, data drift, changing models, running models on devices, updating those models, choosing model size, performance in non-English languages, tokenization cost, and whether chat is an efficient interface. She closes with data scarcity. Public training data may run out, while the web gains more AI-generated material. Her practical conclusion is that companies need to understand and govern their own data before relying on generative AI.

## Key ideas
### LLM applications are difficult to make consistent
[04:01](https://www.youtube.com/watch?v=spamOhG7BOA&t=241s)
Huyen begins with consistency. Users expect an application to behave predictably, and downstream software often needs to process the model's response. The same input can produce different outputs because of the stochastic nature of language models. Setting temperature to zero does not solve every case. Even a small change in the input can lead to a completely different answer, as in her example asking ChatGPT to give a review score from one to five. This becomes a practical engineering problem when an application needs to parse a score or another structured value, because there is no reliable way to enforce an output schema.

### Hallucinations block high-stakes adoption
[05:38](https://www.youtube.com/watch?v=spamOhG7BOA&t=338s)
Huyen calls hallucination one of the biggest barriers to company adoption. She gives an example of a teacher who used ChatGPT to write essays and asked students to correct them, finding hallucinated information in every essay. The risk is especially serious for factual tasks, legal work, contract processing, and code generation. She says language models perform poorly on text-to-SQL, with the best model on the benchmark she shows achieving under 50 percent accuracy. She discusses hypotheses involving weak cause-and-effect understanding and a mismatch between the model's internal knowledge and the knowledge of the people who label training answers.

### Privacy depends on whether the model is built or bought
[08:06](https://www.youtube.com/watch?v=spamOhG7BOA&t=486s)
Privacy creates different responsibilities for in-house systems and external APIs. A chatbot connected to company data must not reveal personally identifiable information by accident. Huyen points to jailbreak examples and says API providers such as OpenAI spend substantial effort preventing unintended disclosure. When a company builds the chatbot itself, that company owns the protection problem. When it buys access, it depends on the provider's compliance practices. She notes that OpenAI retained data for 30 days at the time of the talk, and says whether that is acceptable depends on the industry and the sensitivity of the application.

### Long context does not remove the need for context design
[09:07](https://www.youtube.com/watch?v=spamOhG7BOA&t=547s)
Huyen argues that context learning will remain necessary because many questions depend on where and how they are asked. A question about the best Chinese restaurant has a different answer depending on the user's location. She cites research on situated question answering that found roughly 16.5 percent of questions require context. Document processing, summarization, and work involving genes or proteins can require very long inputs, sometimes reaching hundreds of thousands of tokens. A model accepting that much text does not prove it can use all of it effectively. Longer context also raises questions about efficiency and cost.

### Models and data drift make tested prompts age quickly
[11:03](https://www.youtube.com/watch?v=spamOhG7BOA&t=663s)
Generative AI has made data drift visible to a wider audience. Huyen describes models trained on older data failing to answer questions about current events even when the relevant evidence is supplied in the context, with a performance drop of about 15 percent in the study she cites. Applications also face model drift when a company replaces its underlying model or when a new architecture or training technique appears. A prompt that works with the current model may behave differently with the replacement. She says there has been little research on how well tested prompts transfer across models.

### On-device LLMs trade privacy for difficult update problems
[12:38](https://www.youtube.com/watch?v=spamOhG7BOA&t=758s)
Running an LLM on a device can keep sensitive information local and avoid dependence on an unreliable internet connection. Huyen connects this need to healthcare centers, autonomous vehicles, and the idea of a personal model running on a user's MacBook. The device must have enough compute for inference, and techniques such as quantization may be needed to make that practical. Updating the model is harder. The world changes, new models appear, and on-device training is limited by compute and memory. Training on a server creates a privacy problem if data must be sent away from the device.

### Model size and language coverage affect production cost
[15:55](https://www.youtube.com/watch?v=spamOhG7BOA&t=955s)
Choosing a model requires more than counting parameters. Huyen describes a tradeoff between performance and cost, and says the best point changes over time. A sparse model with 100 billion parameters can behave differently from a dense model with the same nominal scale, so parameter count alone is not enough. She also discusses poor performance in many non-English and low-resource languages. Tokenization can produce many more tokens for the same input, which raises latency and cost when APIs charge by token. She gives Vietnamese as a personal example and mentions research comparing token lengths across languages.

### Data ownership becomes harder as public training data runs out
[23:35](https://www.youtube.com/watch?v=spamOhG7BOA&t=1415s)
Huyen says models still have capacity to learn from more data, while the supply of newly generated public data grows more slowly than the size of training datasets. She cites a study predicting that publicly available internet data could run out around 2026. The problem is compounded by the web filling with text generated by existing AI systems, which means future models may train on material produced by earlier models. For companies, this makes the data story an early priority. She describes organizations consolidating data across departments and sources, then updating terms of use and adding quality and governance controls. The point is practical: a company needs to know what data it owns, what it may use, and how that data changes before it builds a dependable LLM application.

## Notable quotes
- "The same input and different outputs and you can enforce determinism by saying temperature is equal to zero. However it doesn't really fix the cases when you can just have a small changes in the input and it can lead to a very big output changes." (04:40)
- "I don't think the context learning will ever go away." (09:27)
- "A model might be able to take in a hundred thousand tokens as input length but how efficiently can that model use on this token I think this is still open question." (10:44)
- "Data is very essential to any companies that want to leverage AI." (24:14)

## Tools & references mentioned
- ChatGPT
- OpenAI
- Dan Grover
- Designing Machine Learning Systems
- Claypot AI
- Common Crawl
- Reddit
- Vietnam
- Japan
- Alpaca

## Who should watch
- You are building an LLM feature that must feed structured results into ordinary software and need to understand why temperature settings do not guarantee stable behavior.
- Your application handles private data, current information, long documents, or a language with limited representation in model training data.
- You are deciding whether to run a model on devices and need to think through inference limits, updates, training, and data ownership.

## Related talks

- [Fireside Chat - The Future of LLMs](https://mlopstalks.com/talks/fireside-chat-the-future-of-llms) (David Hershey, Unusual Ventures & Daniel Jeffries, AI Infrastructure Alliance, 36:07)
- [Want High Performing LLMs? Hint: It Is All About Your Data](https://mlopstalks.com/talks/want-high-performing-llms-hint-it-is-all-about-your-data) (Vikram Chatterji, Galileo, 33:56)
- [LLMs in Production Conference - Part II](https://mlopstalks.com/talks/llms-in-production-conference-part-ii) (Matei Zaharia, Databricks and Stanford & Chip Huyen, 37:21)
- [The Confidence Checklist for LLMs in Production](https://mlopstalks.com/talks/the-confidence-checklist-for-llms-in-production) (Rohit Agarwal, portkey.ai, 32:34)
- [Building Products](https://mlopstalks.com/talks/building-products) (Sam Charrington, TWIML AI Podcast & George Mathew, Insight Partners & Asmitha Rathis, PromptOps & Natalia Burina, Meta & Sahar Mor, Stripe, 45:18)