# Understanding the LLM Economics

Nikunj Bajaj, TrueFoundry | LLMs in Production 2023 | 31:23

Source: https://www.youtube.com/watch?v=WQJdjRyYPwY
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/understanding-the-llm-economics
Published: 2023-07-10
Tags: cost, fine-tuning, inference, open-models

## TL;DR
- Summarizing all of Wikipedia to half its size with GPT-4 and an 8K context window would cost about $360,000 under the pricing used in the talk.
- Self-hosting a 7-billion-parameter model on an eight-GPU spot instance could reduce the example cost to about $2,100, before other infrastructure expenses.
- Fine-tuning commercial models costs much more than fine-tuning self-hosted open-source models, so smaller fine-tuned models can make sense for simpler tasks.

## Summary
Nikunj Bajaj explains how to estimate the cost of LLM applications through a worked example: summarizing six million Wikipedia articles from about 1,000 tokens each to 500 tokens each. Using the prices available at the time, GPT-4 with an 8K context window costs about $360,000, while the 32K version costs about $720,000. Anthropic's model costs about $162,000, and cheaper OpenAI models reduce the figure further. A self-hosted 7-billion-parameter model on an eight-GPU spot instance costs about $2,100 for the same processing example. Fine-tuning changes the comparison. Commercial fine-tuning can push the cost above $1.25 million, while fine-tuning the self-hosted model brings the example to about $3,500. Bajaj argues that commercial and open-source models will coexist. Smaller models can handle specific, simpler tasks, while larger models are better suited to ambiguous reasoning and long contexts. He also describes TrueFoundry's token compression and model deployment tools.

## Key ideas
### The talk uses Wikipedia summarization to make LLM pricing concrete
[03:59](https://www.youtube.com/watch?v=WQJdjRyYPwY&t=239s)
Bajaj defines a simple calculation: summarize six million Wikipedia articles, each approximated at 750 words or 1,000 tokens, to half their size. The output would contain about 500 tokens per article. He says the figures are approximations, so the exercise is meant to build intuition rather than provide an exact bill. The calculation focuses only on cost. Quality, latency, privacy, and other model-selection concerns remain outside this example.

### API pricing charges separately for prompt tokens and response tokens
[05:51](https://www.youtube.com/watch?v=WQJdjRyYPwY&t=351s)
For GPT-4 with an 8K context window, Bajaj uses a price of $30 per million prompt tokens and $60 per million response tokens. The six million article inputs therefore cost about $180,000. Producing half as many output tokens costs another $180,000. Together, the example reaches the talk's $360,000 figure. The expense comes from both the material sent to the model and the material generated by it.

### Larger context windows and models raise the bill quickly
[07:42](https://www.youtube.com/watch?v=WQJdjRyYPwY&t=462s)
Bajaj says the GPT-4 model with a 32K context window uses prices that are twice those of the 8K version, bringing the example to about $720,000. He gives an Anthropic example at about $162,000, based on different prompt and response prices. Cheaper OpenAI models reduce the cost further, with his example reaching about $18,000 for Curie. His comparison shows why model choice and context length need to be part of the application design.

### Self-hosting changes the unit of cost from tokens to GPU time
[09:24](https://www.youtube.com/watch?v=WQJdjRyYPwY&t=564s)
For a self-hosted model, Bajaj calculates cost from GPU time rather than API token prices. He uses an eight-GPU spot instance priced at $10 per hour and a roughly seven-billion-parameter model. Inputs can be processed in parallel, while outputs are generated sequentially and take longer. In his calculation, processing all the inputs costs about $350 and generating the outputs costs about $1,750, for a total of roughly $2,100.

### Fine-tuning commercial models can be far more expensive than fine-tuning open-source models
[11:47](https://www.youtube.com/watch?v=WQJdjRyYPwY&t=707s)
Fine-tuning the commercial DaVinci model changes its token prices to $120 per million for both inputs and outputs. Bajaj estimates $720,000 for processing the inputs, $360,000 for the outputs, and a total above $1.25 million after adding fine-tuning. The cheaper Curie example comes to about $126,000. By comparison, fine-tuning the self-hosted model leaves its processing costs unchanged and brings the total example to about $3,500.

### A small fine-tuned model can handle a specific task
[16:19](https://www.youtube.com/watch?v=WQJdjRyYPwY&t=979s)
Bajaj cites work from Moveworks that evaluated a fine-tuned seven-billion-parameter model across 14 use cases. In the examples he presents, the smaller model reaches performance close to GPT-4 few-shot learning and exceeds it on some tasks. He uses this to argue that open-source and commercial models can coexist inside the same application. A smaller model can handle a narrow task with suitable fine-tuning, while a larger model can handle ambiguous reasoning and very long context.

### Prompt compression can reduce the input portion of API costs
[18:43](https://www.youtube.com/watch?v=WQJdjRyYPwY&t=1123s)
Bajaj says prompt processing accounts for half or more of the total cost in many of his examples. His proposed compression API removes context that the model does not need while aiming to preserve the answer. In a Beyonce example, the original context uses 649 tokens, while the compressed version uses 465 tokens and returns the same answer. He presents this as roughly a 30 percent reduction in token use, although he says the approach does not work equally well for every task.

### Compression quality depends on the task and may require fine-tuning
[26:09](https://www.youtube.com/watch?v=WQJdjRyYPwY&t=1569s)
In the discussion, Bajaj says simply removing stop words or reducing words without preserving context makes model performance drop. Few-shot examples improve the result, but his stronger approach fine-tunes a model for the compression task. He is clear that this is not general-purpose. Compressing code, for example, performs poorly in his experience. He also says seven-billion- and 13-billion-parameter models are not generally as capable as larger commercial models, but they can work well for simpler classification or other focused tasks.

## Notable quotes
- "This is going to be practically a math presentation and we are going to be dealing with a lot of numbers." (03:38)
- "That's your number three hundred sixty thousand dollars to take Wikipedia and reduce it to half the size." (07:21)
- "The way you price self-hosted models is actually very very different compared to how you would price the API calls." (09:24)
- "Eventually where we got to much better performance was actually fine tuning one of the LLMs with this specific task that you learn to not lose context but still reduce the number of tokens." (27:01)

## Tools & references mentioned
- TrueFoundry
- GPT-4
- OpenAI
- Anthropic
- DaVinci
- Curie
- Moveworks
- FrugalGPT
- GPT-3
- Dolly
- MPT
- Hugging Face
- Kubernetes

## Who should watch
- You are estimating the cost of adding LLM features and need a concrete way to separate prompt, response, GPU, and fine-tuning costs.
- You are deciding between commercial APIs and self-hosted open-source models for tasks with different levels of complexity.
- You are considering prompt compression or smaller fine-tuned models and want to understand where those approaches can fail.

## Related talks

- [Making LLM Inference Affordable](https://mlopstalks.com/talks/making-llm-inference-affordable) (Daniel Campos, Snowflake, 32:07)
- [Cost Optimization and Performance](https://mlopstalks.com/talks/cost-optimization-and-performance) (Lina Weichbrodt & Luis Ceze, OctoML & Jared Zoneraich, Prompt Layer & Daniel Campos, Neeva & Mario Kostelac, Intercom, 36:06)
- [Exploring the Latency/Throughput & Cost Space for LLM Inference](https://mlopstalks.com/talks/exploring-the-latency-throughput-cost-space-for-llm-inference) (Timothée Lacroix, Mistral, 30:25)
- [LLMs vs LMs in Production](https://mlopstalks.com/talks/llms-vs-lms-in-production) (Denys Linkov, Voiceflow, 24:44)
- [The Truth About LLM Training](https://mlopstalks.com/talks/the-truth-about-llm-training) (Paul van der Boor & Zulkuf Genc, Prosus Group, 55:47)