LLM applications can reduce inference costs by adapting prompts, approximating model responses, and routing queries through a cascade of models.
2
FrugalGPT combines prompt concatenation, caching, and adaptive model selection to avoid repeated work and expensive API calls.
3
A cascade uses a scoring model to decide whether a cheaper model has produced an acceptable answer before calling a more expensive model.
Summary
Lingjiao Chen explains how FrugalGPT reduces the cost of applications that call large language models. He frames hosted LLMs as providers in a marketplace, with large differences in price and capability. The approach has three parts: prompt adaptation, model approximation, and model cascades. Prompt concatenation lets several questions share the same few-shot examples, while caching avoids processing queries that are repeated or sufficiently similar. A cascade sends a query through models in order of cost and uses a separate model to score each answer. The system stops when an answer passes a quality threshold, so it does not call a more expensive model unnecessarily. Chen also describes an automated process for choosing which APIs belong in a cascade for a particular application. The methods can be combined. He is honest that cache eviction and semantic similarity remain application-dependent problems, especially when superficially similar questions have opposite meanings.
Chen connects machine learning systems to data management through their shared focus on data
Lingjiao Chen says his database background led him to see a common problem in data management and machine learning: how to process data efficiently and in a user-friendly way. Machine learning has specialized hardware and algorithms, but without data those tools cannot do much. He describes data management as another way to view machine learning research, engineering, and deployment. It can provide abstractions for organizing data around machine learning workloads. Chen also says data management is broader than relational tables. It has included graph, image, audio, and multimedia data, so the field can inform systems that handle many data types.
FrugalGPT treats hosted language models as providers in a marketplace
Chen draws a comparison with the idea of a sky abstraction built over multiple cloud providers. In the same way, FrugalGPT lets an application operate over several language model services without depending on each provider's internal details. He also connects this to data marketplaces. If data can include predictions produced by machine learning services, then a marketplace of language model APIs can be viewed as another form of data market. Demetrios Brinkmann restates the practical implication: OpenAI or another provider can be one source among several, rather than the only system used by an application.
Prompt concatenation shares repeated examples across several questions
Chen describes a prompt adaptation method that combines several queries into one request. Suppose two questions each include the same few-shot question-and-answer examples. Sending them separately makes the model process those examples twice. A concatenated request includes the examples once, followed by both questions and identifiers for their corresponding answers. Chen says the identifiers keep the outputs separate. In the example shown, a chemistry question and a nature question are answered in one request. The saving comes from avoiding redundant prompt processing, while the application still receives a distinct answer for each question.
Prompt concatenation reached four questions and about 70 percent lower cost in Chen's test
When Demetrios asks how far the method can be pushed, Chen says the team tested up to four questions. He reports that this saved about 70 percent of the cost per request without hurting accuracy. He adds that the limit depends on the model's maximum prompt size. If each prompt uses a few hundred tokens, a larger context window could allow many more examples or questions to be combined. At the time of the conversation, the process required identifying suitable combinations, and Chen said code for FrugalGPT would be released so users could apply the method to their own applications.
A model cascade stops at the first answer that meets the quality threshold
Chen's second approach uses a sequence of language model APIs ordered roughly by cost. A query goes to the first model. If its answer is not good enough, the next model receives the query, and the process continues until an answer meets the application's quality requirement. The expensive final model is called only when cheaper models fail. The difficult part is judging answer quality automatically. Chen describes an AI model that takes the query and a candidate answer as input and produces a quality score. A threshold then determines whether the cascade should stop. FrugalGPT can select a different set of APIs for different applications or datasets.
Caching can remove repeated LLM calls, although similarity is difficult to define
Chen describes model approximation through a cache layer. Before sending a query to a language model, the application checks whether it has already processed the same or a similar query. If it has, the stored answer can be returned without another model call. This can help applications such as search or customer service when users ask similar questions during a period of time. The cache introduces practical decisions about which entries to retain when memory is limited. Similarity is also risky. Questions that look alike can have opposite meanings, such as asking what happens when substances are combined versus asking what does not happen. Chen says there is no general solution yet, so the right approach depends on the application.
Caching, prompt concatenation, and cascades can be composed
Chen says the methods do not have to be used separately. A cache can store answers from the services in a cascade, allowing a later similar query to avoid the cascade entirely. Cached outputs from a concatenated batch can also be reused. The methods can be combined in the other direction as well: an application could first concatenate a large set of queries into smaller batches and then send those batches through a model cascade. Chen gives the example of turning 1,000 queries into 500 combined requests before applying the cascade. The resulting design depends on the application's query patterns, model costs, and quality requirements.
"I want to work on things that are mathematically profound but also practically useful, so that other people would be able to benefit from this kind of research."Lingjiao Chen08:55
Who should watch
You are paying for hosted LLM calls and need to reduce spend without automatically sending every query to the most expensive model.
Your application has repeated or closely related queries, and you want to understand where caching can remove model calls.
You are designing a multi-model LLM service and need a practical explanation of routing, answer scoring, prompt batching, and their limits.
