# DSPy Assertions: Computational Constraints for Self-Refining LM Pipelines

Arnav Singhvi, DSPy | MLOps Community | 38:26

Source: https://www.youtube.com/watch?v=3_ygPgTdowE
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/dspy-assertions-computational-constraints-for-self-refining-lm-pipelines
Published: 2024-04-24
Tags: guardrails, prompt-engineering

## TL;DR
- DSPy replaces manually written prompt templates with programs built from signatures, modules, and optimizers.
- DSPy Assertions let developers express constraints in Python, then trigger self-correction or backtracking when a language model violates them.
- Assertions improved formatting, citation faithfulness, distractor quality, and tweet-generation measures across the tasks Arnav Singhvi presented.

## Summary
Arnav Singhvi presents DSPy as a way to program language models through reusable signatures, modules, and optimizers instead of maintaining long prompt templates. A DSPy program declares inputs and outputs, combines prompting techniques such as Chain of Thought and retrieval, and can be compiled with examples and a metric to produce an optimized prompt. The talk then introduces DSPy Assertions, which connect ordinary validation functions to language model calls. A strict assertion can halt execution after failed attempts, while a suggestion logs the failure and continues after asking the model to refine its output. Singhvi shows constraints for query length, query diversity, citations, JSON formatting, plausible quiz distractors, hashtags, answer correctness, engagement, and faithfulness. The examples show gains on multi-hop QA, long-form QA, quiz generation, and tweet generation. He is also direct about trade-offs: assertions add model calls and can increase token use, although compilation and multithreading can reduce the cost or latency in some setups.

## Key ideas
### DSPy turns prompting techniques into programmable modules
[01:02](https://www.youtube.com/watch?v=3_ygPgTdowE&t=62s)
Singhvi says DSPy is built around programming language models rather than manually tweaking prompts. Its signatures declare inputs and outputs, such as a question and an answer or a document and a summary. Its modules package techniques such as Chain of Thought and ReAct so they can be combined in a pipeline. Its optimizers take a DSPy program, examples, and a metric, then search for prompt instructions and demonstrations that improve the metric. This structure replaces long, hand-written templates with a program whose parts can be changed and compiled.

### DSPy compilation teaches smaller models from selected examples
[08:50](https://www.youtube.com/watch?v=3_ygPgTdowE&t=530s)
The DSPy compiler runs a declared program over examples, evaluates the results with a supplied metric, and retains examples that show good behavior. Those examples become demonstrations in an optimized prompt. Singhvi describes using a larger model such as GPT-4 to generate useful behavior, then giving the resulting demonstrations to a smaller model such as GPT-3.5. In the presented multi-hop setting, answer exact match rose from 34.3% to 54.7% for one setup, while Llama 2 moved from 27.5% to 50%. He also mentions bootstrap few-shot, bootstrap fine-tuning, and signature optimization as different compilation strategies.

### Assertions express model constraints through ordinary validation code
[12:02](https://www.youtube.com/watch?v=3_ygPgTdowE&t=722s)
DSPy Assertions add programmatic checks around language model outputs. A developer supplies a validation function, the model output, and feedback that explains the constraint. A strict dsp.assert stops execution when the output keeps failing after the allowed attempts. A dsp.suggest records an unresolved failure and lets the program continue, which makes it useful during broader evaluation. When a check fails, DSPy adds the prior output and feedback to the signature and retries the module. The model therefore receives information about what it produced and the specific condition it violated.

### Backtracking can make multi-hop retrieval queries shorter and distinct
[14:53](https://www.youtube.com/watch?v=3_ygPgTdowE&t=893s)
Singhvi applies assertions to a multi-hop question-answering program. One suggestion checks that a generated query has fewer than 100 characters. Another checks that the query differs from earlier queries, so retrieval does not repeatedly return the same passages. If a query is too long, DSPy modifies the module signature with the previous query and the feedback, then tries again. In his John Loeb example, the first query repeats much of the question. The refined behavior searches for a shorter term such as the University of Maryland, which is present in the context and more useful to the retriever.

### Assertions can check qualities that require a model judge
[17:07](https://www.youtube.com/watch?v=3_ygPgTdowE&t=1027s)
The evaluation goes beyond simple question answering. In long-form QA, the output must place citations after every one or two sentences and keep each citation faithful to the source. In quiz generation, the output must use JSON, include the correct answer, and produce plausible distractors. Singhvi says the citation-faithfulness measure increased by about 13 to 14 percent with assertions. Quiz validity rose by about 40 to 45 percent. These checks include easy programmatic conditions such as formatting and more subjective conditions such as faithfulness and plausibility, which can be evaluated by another language model.

### Assertions improve outputs with many simultaneous constraints
[22:26](https://www.youtube.com/watch?v=3_ygPgTdowE&t=1346s)
Tweet generation combines several requirements: the answer must be correct, the tweet must fit the character limit, hashtags should be omitted, and the text should be engaging and faithful to its context. Without assertions, a tweet can satisfy some conditions while failing others. Singhvi reports that the no-hashtag measure rose from 21% to 71%, engagement rose from 1% to 90.7%, faithfulness rose from 63% to 75%, and overall quality increased by 14 points. His comparison shows that the refined tweet keeps the answer and length while also removing hashtags and better matching the source.

### Assertions add calls and cost, although compilation can shift work away from inference
[29:08](https://www.youtube.com/watch?v=3_ygPgTdowE&t=1748s)
In the question-and-answer portion, Singhvi acknowledges that assertions can add language model calls and make a pipeline token-heavy. Compilation with a larger model searches for a limited number of good demonstrations, while the smaller inference model handles the later requests. Multi-threaded settings can reduce the effect of extra calls on latency. Developers can also choose cheaper DSPy strategies such as bootstrap few-shot rather than searching across many candidates, although Singhvi says that may reduce performance. For simple checks such as length and formatting, ordinary Python functions are enough; more intangible checks depend on a judging model and are less reliable.

## Notable quotes
- "DSPy's core philosophy lies on the fact of programming not prompting models." (04:28)
- "Assertions in DSPy are Python that help you understand where your program is going wrong." (13:01)
- "We're not trying to do more than that, we're just trying to give it an informed response of what to fix." (28:40)
- "Pipelines not prompts." (37:35)

## Tools & references mentioned
- DSPy
- Chain of Thought
- retrieval-augmented generation
- ReAct
- HotpotQA
- PyTorch
- GPT-4
- GPT-3.5
- Llama 2 13B Chat
- GPT-3.5 Turbo
- Wikipedia
- Databricks
- Baseten
- MLOps Community

## Who should watch
- You maintain long prompt templates and want a programmatic way to compose language model calls and optimize their instructions.
- Your pipeline needs outputs to obey checks for formatting, length, citations, correctness, or retrieval behavior.
- You are weighing self-refinement against added model calls and want to understand how compilation can move some work away from inference.

## Related talks

- [DSPy: Transforming Language Model Calls into Smart Pipelines](https://mlopstalks.com/talks/dspy-transforming-language-model-calls-into-smart-pipelines) (Omar Khattab, Stanford, 1:05:40)
- [Beyond Guess-and-Check: Towards AI-assisted Prompt Engineering](https://mlopstalks.com/talks/beyond-guess-and-check-towards-ai-assisted-prompt-engineering) (Alex Cabrera, Carnegie Mellon University, 17:45)
- [The Emerging Toolkit for Reliable, High-quality LLM Applications](https://mlopstalks.com/talks/the-emerging-toolkit-for-reliable-high-quality-llm-applications) (Matei Zaharia, Databricks, 31:01)
- [Harnessing AI APIs for Safer, Accurate, & Reliable Applications](https://mlopstalks.com/talks/harnessing-ai-apis-for-safer-accurate-reliable-applications) (Ron Heichman, SentinelOne, 1:08:14)
- [Ax a New Way to Build Complex Workflows with LLMs](https://mlopstalks.com/talks/ax-a-new-way-to-build-complex-workflows-with-llms) (Vikram Rangnekar, Stealth, 50:51)