DSPy shifts language model development from hand-written prompt strings toward declarative modules arranged in ordinary code.
2
Its compiler can optimize module parameters, demonstrations, and fine-tuning choices against a metric and a dataset.
3
Omar Khattab argues that most deployed applications need carefully designed workflows rather than general-purpose agents.
Summary
Omar Khattab explains why DSPy treats language model applications as programs instead of collections of hand-tuned prompts. He traces the project from an earlier abstraction built around user-created demonstrations to DSPy, where users define modules, signatures, workflows, data, and metrics while the compiler searches for effective prompting or fine-tuning configurations. The same program can compile into a short prompt for GPT-4, a few-shot prompt for Llama 2, or a fine-tuned smaller model. Omar connects this approach to retrieval systems, especially multi-hop retrieval and late interaction, where documents are represented with many token-level vectors rather than one document vector. He also argues that agents are ordinary workflows with loops and tool calls. They can be compiled in DSPy, but most production tasks do not require them. The practical message is to improve the system architecture and its measurable objective before changing prompts or reaching for fine-tuning.
DSPy emerged after the team rejected an abstraction that made users design the compiler
Omar explains that the first version of DSP was built as an abstraction layer for language model pipelines, but users were not yet building many such pipelines. In version one, the difficult part was the word "demonstrate": users had to create the demonstrations and the compiler-like process themselves. That proved confusing and did not deliver the value the team wanted. After returning to the drawing board, they decided that DSPy should provide the compilers and pipeline modules, while users write the workflow. DSPy was released in August 2023. This change moved complexity away from the person building the application and into an open-source optimization system.
Retrieval quality depends on the whole workflow, not only the embedding model
Omar distinguishes the common modern meaning of RAG from the wider field of retrieval-based NLP. A basic system retrieves passages from a vector database and gives them to a language model. Harder tasks require multi-hop retrieval, query generation, claim checking, result aggregation, and document comparison. He describes late interaction as a different retrieval representation: documents are stored as matrices of small token-level vectors rather than one vector per document. The search infrastructure remains, but the encoder can represent different parts of a document more precisely. Omar says this can improve quality in difficult domains with little training data, while keeping search close to the speed and efficiency of single-vector encoding.
Hand-written prompts become brittle as models, domains, and pipeline stages change
Omar says researchers commonly build language model pipelines by writing prompt strings, editing them until they work for a favorite model, and chaining the pieces together. This breaks when the model changes, when questions become harder, or when the domain changes. It also breaks inside multi-stage pipelines because an intermediate prompt has no direct labels and is mainly trying to connect earlier and later stages. DSPy addresses this by replacing open-ended prompt editing with modules that have defined inputs and outputs and parameters that can be optimized. A module might take a question and paragraphs and return an answer. The developer can reuse it in loops or other control flow, then compile the whole program against data and a metric.
DSPy modules describe behavior through natural-language signatures
DSPy modules are more general than prebuilt task-specific wrappers around fixed prompts. Omar compares them to reusable neural network layers. A Chain of Thought module says that a model should solve a subtask step by step. A Program of Thought module says that the model should generate and execute code, then interpret the result. The module receives a signature such as "long document" to "summary" or "question and paragraphs" to "response." The signature specifies input and output behavior without requiring a long prompt. Developers can compose modules into ordinary programs with loops, recursion, exceptions, and other control flow. Language models handle fuzzy computations such as questions, answers, passages, and SQL queries inside that larger program.
The compiler can turn one DSPy program into different prompting or fine-tuning strategies
After a program is written, the compiler executes it on examples and tracks successful traces through its modules. A metric determines whether an output is useful, such as answer correctness or a criterion evaluated by another DSPy program. The compiler can collect effective traces as demonstrations and use them in later prompts. Different teleprompters can instead fine-tune smaller models or use other optimization methods. Omar gives the example of compiling the same program into a zero-shot prompt for GPT-4, a few-shot prompt for Llama 2, or a fine-tuned model such as Flan-T5. Prompting and fine-tuning become alternative ways to implement a declared module, rather than entirely separate application architectures.
Breaking a task into modules can reduce the need for the most expensive model
Omar addresses the concern that a modular pipeline might create a large language model bill. He says individual steps usually need far less capability than the full task, and a typical pipeline has at most one call that needs a very powerful model. The other steps may break down a problem, combine information, or perform another limited operation. Compilation can also produce pipelines using much smaller models. He mentions Flan-T5 large, with 700 million parameters, as an example of a model that can handle individual compiled steps. Because the model is optimized for a specific subtask and the program is evaluated on the developer's data, the result can cost less than repeatedly calling a large general-purpose model.
Fine-tuning is a replaceable implementation choice inside the system
Omar argues that a fine-tuned model is only one component of an application. The system remains the program, while one or more models process particular modules. In DSPy, changing from few-shot prompting to fine-tuning does not require throwing away the workflow. The developer can switch the teleprompter and test another implementation. This gives teams a structured way to respond when a pipeline is too expensive, makes a consistent mistake, or needs to handle an edge case. The alternatives include changing the module, adding an exception in code, or changing the metric so that the compiler selects better behavior. Omar is direct that fine-tuning does not remove the need to gather and prepare useful data.
Agents are workflows with loops, so most production systems should start with simpler programs
Omar describes agents as programs that usually contain a loop and modules that decide when to call tools. Since they are ordinary workflows, DSPy can compile them like other programs. He is more cautious about using them in deployment. The community has not fully worked out how to solve many well-defined tasks, yet often jumps straight to general-purpose agents. Omar says that 90 percent of deployed applications do not need agents and will perform better without them. Agents still have value for exploration and research. When building one, he recommends selecting modules, connecting them in code, and compiling against the actual goals rather than writing a collection of prompts.
"Our solution to that is sort of keeping the focus on building these architectures and building these pipelines, but shifting it from kind of a hand tuning game of open-ended free form prompts to a more structured and automatically optimizable view of the world."Omar Khattab32:29
Who should watch
You maintain an LLM or RAG application whose prompts need regular manual edits when the model or domain changes.
You want to compare few-shot prompting, fine-tuning, and smaller models within the same application workflow.
You are considering an agent and need a case for starting with a more constrained, measurable program.
