The LLM application stack is forming around foundation models, vector databases, developer tools, and monitoring systems.
2
Reliable LLM applications need model comparison, output tracking, deployment, monitoring, and evaluation, just as traditional ML applications do.
3
DSP lets developers describe a multi-step LLM pipeline while the system searches for prompts, retrieval strategies, and model configurations that improve its results.
Summary
Demetrios Brinkmann opens with findings from the LLMs in Production survey. He describes an emerging stack around foundation models, vector databases, developer SDKs, model tuning, and monitoring. Survey responses show that cost is a more common concern among people not using OpenAI, while hallucinations remain a serious production risk. Matei Zaharia then explains why fluent LLM output is not enough for production. Models can give incorrect facts, produce code that does not solve the real problem, change behavior over time, and create privacy, cost, latency, and update problems. He describes MLflow work for comparing models and pipelines, tracking prompts and outputs, evaluating generated text, deploying models, and monitoring production data. Zaharia also presents DSP, a declarative programming model that separates pipeline design from prompt and model optimization. DSP can use demonstrations, search, and prediction strategies to tune multi-step systems before deployment. He is open about the tradeoff: more stages can improve quality while adding cost and latency.
An LLM application stack is forming around models, data access, and operations
Demetrios Brinkmann describes a stack with foundation models at the base, vector databases alongside them, model tuning and model building above that, and developer tools such as LlamaIndex and LangChain. At the top are model operations, monitoring, and experimentation, with examples including Portkey, Weights & Biases, and Arize. He says the industry is beginning to see an LLM stack, although the terminology is still unsettled. The conference brings together companies working across these layers, including Redis, Chroma, and Milvus.
Cost and hallucinations are changing which models teams can use in production
Brinkmann says 25% of survey respondents who were not using OpenAI cited cost, compared with 15% of respondents who were using OpenAI. He warns that pricing was changing quickly, so the result was only a snapshot. For teams that had not put LLMs into production, hallucinations were a major concern because incorrect output could damage a company's reputation. He frames the conference as a response to these practical questions rather than as a discussion of model hype.
Interest in fine-tuning, embeddings, inference, and open models points beyond hosted APIs
The survey respondents were especially interested in fine-tuning, embeddings, inference, and working with open-source models. Brinkmann interprets this as evidence that people wanted more control and had not found everything they needed in OpenAI's offering. These interests explain why the conference focuses on the parts of an LLM system that sit around the model itself, including retrieval, model behavior, and serving.
Production LLM applications still need the basic controls from ML systems
Matei Zaharia argues that reliable LLM applications need familiar ML application infrastructure. Teams should be able to swap and compare models or multi-step pipelines, track and evaluate outputs, keep a historical record of runs, deploy pipelines reliably, and monitor deployed applications over time. MLflow is extending its model abstraction and automatic logging to support hosted models, local models, and chains. The aim is to let application code treat different implementations through a common interface.
LLM systems can fail through incorrect facts, bad code, drift, and privacy problems
Zaharia gives several examples of fluent but wrong output. ChatGPT incorrectly answers when Stanford University was founded, misattributes the creation of a research model to Salesforce Research, and recommends parallelizing code that is actually serial. He also describes operational problems such as expensive inference, service outages under load, model drift, stale knowledge, and privacy requests that require information to be removed. A production system therefore needs more than a convincing demo.
MLflow is adding evaluation and monitoring workflows for generated text
Zaharia describes MLflow support for recording prompts and parameters, saving models and chains, and comparing generated outputs side by side on evaluation data. Users can add ratings or import ratings from a labeling service. Programmatic evaluation can use metrics for summarization, toxicity, and other LLM tasks, with room for custom metrics. After deployment, Databricks can store model activity in a table where SQL-based alerts can detect conditions that need attention.
DSP separates the design of an LLM pipeline from its optimization
Zaharia presents DSP, short for Demonstrate, Search, Predict, as a declarative programming model for LLM pipelines. A program describes components such as language models, retrievers, and other functions, with text passed between them. Demonstrate supplies examples or constraints, Search retrieves useful information, and Predict selects ways to use and check that information. DSP then uses the whole pipeline to select prompting, retrieval, and reasoning strategies, instead of forcing developers to tune each stage manually.
DSP can improve multi-step retrieval systems before deployment, at a cost
In the Baleen example, DSP repeatedly generates a search question, retrieves passages, summarizes the current context, and eventually produces an answer. With labeled examples, DSP can compile the program into a version that uses effective few-shot demonstrations, and it can also fine-tune models for individual steps. Zaharia says this approach performed well on knowledge-intensive research tasks with relatively small example sets. He also acknowledges that adding multiple stages increases cost and latency, and that DSP's optimization is performed offline before deployment.
"We want developers to be able to step away from this and just think about their system design and components, not about gluing and optimizing all these stages together."Matei Zaharia20:36
Who should watch
You are building an LLM application and need a concrete checklist for comparing models, evaluating generated text, deploying pipelines, and monitoring them.
Your system uses retrieval or several model calls, and you want to understand how DSP separates pipeline design from prompt and model optimization.
You are deciding whether a multi-step LLM workflow is worth its added cost and latency, especially for knowledge-intensive questions.
