# LLMs in Production Conference - Part II

Matei Zaharia, Databricks and Stanford & Chip Huyen | LLMs in Production 2023 | 37:21
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=3XNUxwk1M6Y
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/llms-in-production-conference-part-ii
Published: 2023-06-16
Tags: evals, observability, prompt-engineering, rag, reranking

## TL;DR
- The LLM application stack is forming around foundation models, vector databases, developer tools, and monitoring systems.
- Reliable LLM applications need model comparison, output tracking, deployment, monitoring, and evaluation, just as traditional ML applications do.
- DSP lets developers describe a multi-step LLM pipeline while the system searches for prompts, retrieval strategies, and model configurations that improve its results.

## Summary
Demetrios Brinkmann opens with findings from the LLMs in Production survey. He describes an emerging stack around foundation models, vector databases, developer SDKs, model tuning, and monitoring. Survey responses show that cost is a more common concern among people not using OpenAI, while hallucinations remain a serious production risk. Matei Zaharia then explains why fluent LLM output is not enough for production. Models can give incorrect facts, produce code that does not solve the real problem, change behavior over time, and create privacy, cost, latency, and update problems. He describes MLflow work for comparing models and pipelines, tracking prompts and outputs, evaluating generated text, deploying models, and monitoring production data. Zaharia also presents DSP, a declarative programming model that separates pipeline design from prompt and model optimization. DSP can use demonstrations, search, and prediction strategies to tune multi-step systems before deployment. He is open about the tradeoff: more stages can improve quality while adding cost and latency.

## Key ideas
### An LLM application stack is forming around models, data access, and operations
[00:00](https://www.youtube.com/watch?v=3XNUxwk1M6Y&t=0s)
Demetrios Brinkmann describes a stack with foundation models at the base, vector databases alongside them, model tuning and model building above that, and developer tools such as LlamaIndex and LangChain. At the top are model operations, monitoring, and experimentation, with examples including Portkey, Weights & Biases, and Arize. He says the industry is beginning to see an LLM stack, although the terminology is still unsettled. The conference brings together companies working across these layers, including Redis, Chroma, and Milvus.

### Cost and hallucinations are changing which models teams can use in production
[01:32](https://www.youtube.com/watch?v=3XNUxwk1M6Y&t=92s)
Brinkmann says 25% of survey respondents who were not using OpenAI cited cost, compared with 15% of respondents who were using OpenAI. He warns that pricing was changing quickly, so the result was only a snapshot. For teams that had not put LLMs into production, hallucinations were a major concern because incorrect output could damage a company's reputation. He frames the conference as a response to these practical questions rather than as a discussion of model hype.

### Interest in fine-tuning, embeddings, inference, and open models points beyond hosted APIs
[02:22](https://www.youtube.com/watch?v=3XNUxwk1M6Y&t=142s)
The survey respondents were especially interested in fine-tuning, embeddings, inference, and working with open-source models. Brinkmann interprets this as evidence that people wanted more control and had not found everything they needed in OpenAI's offering. These interests explain why the conference focuses on the parts of an LLM system that sit around the model itself, including retrieval, model behavior, and serving.

### Production LLM applications still need the basic controls from ML systems
[11:11](https://www.youtube.com/watch?v=3XNUxwk1M6Y&t=671s)
Matei Zaharia argues that reliable LLM applications need familiar ML application infrastructure. Teams should be able to swap and compare models or multi-step pipelines, track and evaluate outputs, keep a historical record of runs, deploy pipelines reliably, and monitor deployed applications over time. MLflow is extending its model abstraction and automatic logging to support hosted models, local models, and chains. The aim is to let application code treat different implementations through a common interface.

### LLM systems can fail through incorrect facts, bad code, drift, and privacy problems
[08:08](https://www.youtube.com/watch?v=3XNUxwk1M6Y&t=488s)
Zaharia gives several examples of fluent but wrong output. ChatGPT incorrectly answers when Stanford University was founded, misattributes the creation of a research model to Salesforce Research, and recommends parallelizing code that is actually serial. He also describes operational problems such as expensive inference, service outages under load, model drift, stale knowledge, and privacy requests that require information to be removed. A production system therefore needs more than a convincing demo.

### MLflow is adding evaluation and monitoring workflows for generated text
[14:46](https://www.youtube.com/watch?v=3XNUxwk1M6Y&t=886s)
Zaharia describes MLflow support for recording prompts and parameters, saving models and chains, and comparing generated outputs side by side on evaluation data. Users can add ratings or import ratings from a labeling service. Programmatic evaluation can use metrics for summarization, toxicity, and other LLM tasks, with room for custom metrics. After deployment, Databricks can store model activity in a table where SQL-based alerts can detect conditions that need attention.

### DSP separates the design of an LLM pipeline from its optimization
[18:34](https://www.youtube.com/watch?v=3XNUxwk1M6Y&t=1114s)
Zaharia presents DSP, short for Demonstrate, Search, Predict, as a declarative programming model for LLM pipelines. A program describes components such as language models, retrievers, and other functions, with text passed between them. Demonstrate supplies examples or constraints, Search retrieves useful information, and Predict selects ways to use and check that information. DSP then uses the whole pipeline to select prompting, retrieval, and reasoning strategies, instead of forcing developers to tune each stage manually.

### DSP can improve multi-step retrieval systems before deployment, at a cost
[24:18](https://www.youtube.com/watch?v=3XNUxwk1M6Y&t=1458s)
In the Baleen example, DSP repeatedly generates a search question, retrieves passages, summarizes the current context, and eventually produces an answer. With labeled examples, DSP can compile the program into a version that uses effective few-shot demonstrations, and it can also fine-tune models for individual steps. Zaharia says this approach performed well on knowledge-intensive research tasks with relatively small example sets. He also acknowledges that adding multiple stages increases cost and latency, and that DSP's optimization is performed offline before deployment.

## Notable quotes
- Demetrios Brinkmann: "This is just like a snapshot in time for what people are going through." (02:23)
- Matei Zaharia: "There are definitely challenges with the quality of responses in LLMs, especially if you want a production grade app." (09:15)
- Matei Zaharia: "If you want it to work very reliably, you want a way to improve it, you need to start with the basics." (11:31)
- Matei Zaharia: "We want developers to be able to step away from this and just think about their system design and components, not about gluing and optimizing all these stages together." (20:36)
- Matei Zaharia: "The stuff I showed, the DSP.compile stuff, is all before deployment." (34:43)

## Tools & references mentioned
- MLOps Community
- Demetrios Brinkmann
- Matei Zaharia
- Databricks
- Stanford
- MLflow
- OpenAI
- ChatGPT
- Dolly
- FrugalGPT
- Redis
- Chroma
- Milvus
- LlamaIndex
- LangChain
- Portkey
- Weights & Biases
- Arize
- DSP
- Baleen
- LMQL
- Guardrails
- JSONFormer
- Wikipedia
- Bing
- Google Search
- edX
- Data + AI Summit

## Who should watch
- You are building an LLM application and need a concrete checklist for comparing models, evaluating generated text, deploying pipelines, and monitoring them.
- Your system uses retrieval or several model calls, and you want to understand how DSP separates pipeline design from prompt and model optimization.
- You are deciding whether a multi-step LLM workflow is worth its added cost and latency, especially for knowledge-intensive questions.

## Editor's note

Matei Zaharia argues that reliable LLM applications need teams to compare pipelines, track outputs, evaluate them, and monitor deployed systems over time. ZenML records each pipeline run's steps, inputs, outputs, and code version, so teams can trace an artifact or model back to the data and code that produced it. Its pipeline code can run on different infrastructure through configuration.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [The Emerging Toolkit for Reliable, High-quality LLM Applications](https://mlopstalks.com/talks/the-emerging-toolkit-for-reliable-high-quality-llm-applications) (Matei Zaharia, Databricks, 31:01)
- [Systematically Test and Evaluate Your LLMs Apps](https://mlopstalks.com/talks/systematically-test-and-evaluate-your-llms-apps) (Gideon Mendels, Comet, 1:01:43)
- [Evaluating LLM-based Applications](https://mlopstalks.com/talks/evaluating-llm-based-applications) (Josh Tobin, Gantry, 49:50)
- [Pitfalls and Best Practices: 5 Lessons from LLMs in Production](https://mlopstalks.com/talks/pitfalls-and-best-practices-5-lessons-from-llms-in-production) (Raza Habib, Humanloop, 30:27)
- [Fireside Chat - The Future of LLMs](https://mlopstalks.com/talks/fireside-chat-the-future-of-llms) (David Hershey, Unusual Ventures & Daniel Jeffries, AI Infrastructure Alliance, 36:07)