Reliable LLM applications need the same development discipline as other ML applications, including model comparison, output evaluation, deployment, and monitoring.
2
DSP separates the design of an LLM pipeline from the prompts, demonstrations, retrieval choices, and fine-tuning used to optimize it.
3
Tools such as constrained generation and FrugalGPT can control model outputs or improve the cost and quality of multi-model applications.
Summary
Matei Zaharia describes the practical problems that appear when fluent language models are put into production. They can give incorrect facts, generate unsuitable code, change behavior over time, cost too much, become unavailable under load, and create privacy or timeliness problems. He argues that teams should begin with familiar ML practices: compare models and pipelines, record and evaluate outputs, deploy them reliably, and monitor their behavior after release. He discusses how MLflow is being extended for LLMs, including model abstractions, integrations, automatic logging, side-by-side evaluation, ratings, metrics, deployment, and monitoring through data tables. The talk then focuses on DSP, a declarative programming model from his Stanford group. DSP lets developers describe a pipeline while the framework searches for prompts, demonstrations, retrieval strategies, and fine-tuning approaches that improve it. Zaharia also mentions constrained-generation tools and FrugalGPT as other approaches for controlling outputs and managing cost.
Fluent LLMs still produce factual, technical, and operational errors
Zaharia opens with examples where an LLM gives a nearly plausible answer that is still wrong. It says Stanford University was founded in 1891, confusing the first class with the university's founding. It attributes a model called BERT to Salesforce Research even though Zaharia's group developed it at Stanford. It also suggests parallelizing code with a thread-pool executor when the underlying algorithm is serial. Production systems add other problems. Models can be expensive, go down under load, drift as they change, lack recent information, and create privacy issues when users ask for data to be removed.
LLM applications need ordinary ML development and monitoring practices
Zaharia groups LLM reliability work into extending existing ML application practices, creating new programming models, and using other emerging tools. He says teams should be able to swap and compare models or entire pipelines, evaluate outputs across different metrics, keep a historical record of results, deploy applications reliably, and monitor them after release. These practices let a team see how an application changes over time instead of judging it from a single demo. He describes MLflow as the open-source project where Databricks is implementing these capabilities, with related integration into the Databricks platform.
MLflow treats remote models, local models, and multi-step pipelines through one model abstraction
MLflow can wrap a call to OpenAI, a local model, or a pipeline behind a common model interface. Application and evaluation code can then swap implementations without needing to know how each one works. Zaharia says the same approach supports pipelines, with a focus on LangChain integrations. MLflow's automatic logging records prompts and parameters, and saved models can later be loaded and called through a common interface. Its user interface can run several models on evaluation questions and display their generated text side by side. Teams can add ratings, import ratings from labeling services, and run programmatic metrics such as toxicity and summarization measures.
DSP separates pipeline design from the work of optimizing each stage
DSP, or Demonstrate-Search-Predict, is a declarative programming model for pipelines built from foundation models, retrievers, calculators, and other text-processing functions. Zaharia compares it with PyTorch, where layers are connected and data flows between them. In DSP, the components pass text objects. Demonstrate specifies what the system should do, Search breaks down problems and retrieves useful information, and Predict controls how the retrieved information is used and checked. After a developer defines the stages, DSP can try prompting methods, Chain of Thought, and data-selection strategies across the whole pipeline. The developer can focus on system design rather than manually adjusting every connection.
DSP can use labeled examples to discover better prompts and training paths
In the Baleen example, DSP answers a multi-step question by generating a search query, retrieving passages, summarizing the current context, and repeating the process for a limited number of hops. A basic prompt may work only some of the time. With DSP compile, developers can provide questions and correct final answers without specifying the searches. DSP runs the examples, finds successful search paths, and turns them into few-shot demonstrations for later calls. With enough examples, it can also fine-tune a supervised model for an individual stage, such as generating searches or answering questions. This keeps pipeline specification separate from prompt and model optimization.
DSP improves knowledge-intensive tasks while adding cost and latency
Zaharia says DSP was evaluated on research tasks that require looking up information, using a small number of examples. In the comparison he presents, DSP scores better than the other few-shot methods shown because it searches for ways to optimize the pipeline. He also describes using unlabeled examples to train a cheaper model that performs the same task as a more expensive one. The approach has tradeoffs. Breaking a problem into many model calls adds cost and latency, and DSP's optimization happens offline before deployment. Zaharia says DSP can use GPT-4 as its language model, while its combination of search and answer strategies can outperform asking the language model the question directly on the tasks discussed.
"We want developers to be able to step away from this and just think about their system design and components, not about gluing and optimizing all these stages together."16:16
Who should watch
You are building an LLM application and need a development process that goes beyond prompt experimentation and demos.
Your team is deciding how to evaluate, compare, deploy, and monitor generative applications with MLflow or another MLOps stack.
You are interested in declarative LLM pipelines and want to understand how DSP uses examples and offline compilation to improve them.
