# FrugalGPT: Better Quality and Lower Cost for LLM Applications

Lingjiao Chen, Stanford University | MLOps Podcast | Episode 172 | 1:02:59
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=M9EZ7YM88uQ
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/frugalgpt-better-quality-and-lower-cost-for-llm-applications
Published: 2023-08-22
Tags: caching, cost, inference

## TL;DR
- LLM applications can reduce inference costs by adapting prompts, approximating model responses, and routing queries through a cascade of models.
- FrugalGPT combines prompt concatenation, caching, and adaptive model selection to avoid repeated work and expensive API calls.
- A cascade uses a scoring model to decide whether a cheaper model has produced an acceptable answer before calling a more expensive model.

## Summary
Lingjiao Chen explains how FrugalGPT reduces the cost of applications that call large language models. He frames hosted LLMs as providers in a marketplace, with large differences in price and capability. The approach has three parts: prompt adaptation, model approximation, and model cascades. Prompt concatenation lets several questions share the same few-shot examples, while caching avoids processing queries that are repeated or sufficiently similar. A cascade sends a query through models in order of cost and uses a separate model to score each answer. The system stops when an answer passes a quality threshold, so it does not call a more expensive model unnecessarily. Chen also describes an automated process for choosing which APIs belong in a cascade for a particular application. The methods can be combined. He is honest that cache eviction and semantic similarity remain application-dependent problems, especially when superficially similar questions have opposite meanings.

## Key ideas
### Chen connects machine learning systems to data management through their shared focus on data
[10:36](https://www.youtube.com/watch?v=M9EZ7YM88uQ&t=636s)
Lingjiao Chen says his database background led him to see a common problem in data management and machine learning: how to process data efficiently and in a user-friendly way. Machine learning has specialized hardware and algorithms, but without data those tools cannot do much. He describes data management as another way to view machine learning research, engineering, and deployment. It can provide abstractions for organizing data around machine learning workloads. Chen also says data management is broader than relational tables. It has included graph, image, audio, and multimedia data, so the field can inform systems that handle many data types.

### FrugalGPT treats hosted language models as providers in a marketplace
[16:10](https://www.youtube.com/watch?v=M9EZ7YM88uQ&t=970s)
Chen draws a comparison with the idea of a sky abstraction built over multiple cloud providers. In the same way, FrugalGPT lets an application operate over several language model services without depending on each provider's internal details. He also connects this to data marketplaces. If data can include predictions produced by machine learning services, then a marketplace of language model APIs can be viewed as another form of data market. Demetrios Brinkmann restates the practical implication: OpenAI or another provider can be one source among several, rather than the only system used by an application.

### Prompt concatenation shares repeated examples across several questions
[21:06](https://www.youtube.com/watch?v=M9EZ7YM88uQ&t=1266s)
Chen describes a prompt adaptation method that combines several queries into one request. Suppose two questions each include the same few-shot question-and-answer examples. Sending them separately makes the model process those examples twice. A concatenated request includes the examples once, followed by both questions and identifiers for their corresponding answers. Chen says the identifiers keep the outputs separate. In the example shown, a chemistry question and a nature question are answered in one request. The saving comes from avoiding redundant prompt processing, while the application still receives a distinct answer for each question.

### Prompt concatenation reached four questions and about 70 percent lower cost in Chen's test
[38:28](https://www.youtube.com/watch?v=M9EZ7YM88uQ&t=2308s)
When Demetrios asks how far the method can be pushed, Chen says the team tested up to four questions. He reports that this saved about 70 percent of the cost per request without hurting accuracy. He adds that the limit depends on the model's maximum prompt size. If each prompt uses a few hundred tokens, a larger context window could allow many more examples or questions to be combined. At the time of the conversation, the process required identifying suitable combinations, and Chen said code for FrugalGPT would be released so users could apply the method to their own applications.

### A model cascade stops at the first answer that meets the quality threshold
[41:47](https://www.youtube.com/watch?v=M9EZ7YM88uQ&t=2507s)
Chen's second approach uses a sequence of language model APIs ordered roughly by cost. A query goes to the first model. If its answer is not good enough, the next model receives the query, and the process continues until an answer meets the application's quality requirement. The expensive final model is called only when cheaper models fail. The difficult part is judging answer quality automatically. Chen describes an AI model that takes the query and a candidate answer as input and produces a quality score. A threshold then determines whether the cascade should stop. FrugalGPT can select a different set of APIs for different applications or datasets.

### Caching can remove repeated LLM calls, although similarity is difficult to define
[52:53](https://www.youtube.com/watch?v=M9EZ7YM88uQ&t=3173s)
Chen describes model approximation through a cache layer. Before sending a query to a language model, the application checks whether it has already processed the same or a similar query. If it has, the stored answer can be returned without another model call. This can help applications such as search or customer service when users ask similar questions during a period of time. The cache introduces practical decisions about which entries to retain when memory is limited. Similarity is also risky. Questions that look alike can have opposite meanings, such as asking what happens when substances are combined versus asking what does not happen. Chen says there is no general solution yet, so the right approach depends on the application.

### Caching, prompt concatenation, and cascades can be composed
[58:48](https://www.youtube.com/watch?v=M9EZ7YM88uQ&t=3528s)
Chen says the methods do not have to be used separately. A cache can store answers from the services in a cascade, allowing a later similar query to avoid the cascade entirely. Cached outputs from a concatenated batch can also be reused. The methods can be combined in the other direction as well: an application could first concatenate a large set of queries into smaller batches and then send those batches through a model cascade. Chen gives the example of turning 1,000 queries into 500 combined requests before applying the cascade. The resulting design depends on the application's query patterns, model costs, and quality requirements.

## Notable quotes
- Lingjiao Chen: "I want to work on things that are mathematically profound but also practically useful, so that other people would be able to benefit from this kind of research." (08:55)
- Lingjiao Chen: "The key is the data." (11:42)
- Lingjiao Chen: "You could build a large language model cascade to adaptively select which language model to use for your own application." (42:00)
- Lingjiao Chen: "You can avoid sending all user queries to a language model." (54:27)
- Lingjiao Chen: "There are a lot of ways to combine or compose all those things." (1:00:43)

## Tools & references mentioned
- FrugalGPT
- Frugal ML
- GPT-4
- ChatGPT
- J1-Jumbo
- OpenAI
- Microsoft
- Anthropic
- Cohere
- GPT-3
- GPT-Neo
- Dolly
- Matei Zaharia
- James Zou
- Stanford University
- University of Wisconsin
- McKinsey & Company
- QuantumBlack

## Who should watch
- You are paying for hosted LLM calls and need to reduce spend without automatically sending every query to the most expensive model.
- Your application has repeated or closely related queries, and you want to understand where caching can remove model calls.
- You are designing a multi-model LLM service and need a practical explanation of routing, answer scoring, prompt batching, and their limits.

## Related talks

- [Making LLM Inference Affordable](https://mlopstalks.com/talks/making-llm-inference-affordable) (Daniel Campos, Snowflake, 32:07)
- [The Emerging Toolkit for Reliable, High-quality LLM Applications](https://mlopstalks.com/talks/the-emerging-toolkit-for-reliable-high-quality-llm-applications) (Matei Zaharia, Databricks, 31:01)
- [Cost Optimization and Performance](https://mlopstalks.com/talks/cost-optimization-and-performance) (Lina Weichbrodt & Luis Ceze, OctoML & Jared Zoneraich, Prompt Layer & Daniel Campos, Neeva & Mario Kostelac, Intercom, 36:06)
- [LLMs Mini Summit // MLOps Mini Summit Meetup #3](https://mlopstalks.com/talks/llms-mini-summit-mlops-mini-summit-meetup-3) (Thomas Capelle, Weights & Biases & Boris Dayma, Craiyon & Jonathan Whitaker & Robbie McCorkell, Leap Labs, 1:45:28)
- [It Worked When I Prompted It](https://mlopstalks.com/talks/it-worked-when-i-prompted-it) (Soham Chatterjee, Sleek, 14:29)