# Efficient Serving of LLMs for Experimentation and Production with Fireworks.ai

Dmytro Dzhulgakov, Fireworks.ai | MLOps Community | 11:43

Source: https://www.youtube.com/watch?v=O-UUy0ZiaBI
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/efficient-serving-of-llms-for-experimentation-and-production-with-fireworks-ai
Published: 2023-10-27
Tags: cost, fine-tuning, gpus, model-serving

## TL;DR
- Fine-tuning gives teams more control over model outputs and can reduce serving costs by allowing shorter prompts and smaller task-specific models.
- Teams can run many fine-tuned model variants from a shared base model, then merge useful discoveries into later training runs.
- Fireworks.ai serves model variants based on the same base model on shared GPUs, including different LoRA adapters, to improve utilization and reduce cost.

## Summary
Dmytro Dzhulgakov describes how teams move from experimenting with many LLM variants to serving them in production. He explains why fine-tuning is useful for business tasks: it can make outputs more controllable, shorten prompts, and allow a smaller model to handle a narrow task. Larger organizations can let engineers create separate fine-tuned variants from one base model, test them in parallel, and later combine the best data or changes. The serving problem is that ordinary batching works well for one heavily used model, while dozens of lightly used variants leave GPUs underused. Fireworks.ai addresses this by placing variants built from the same base model on shared hardware and batching requests that differ only in their adapters. Dzhulgakov also mentions inference optimizations, OpenAI-compatible access, LangChain integration, and free developer credits. The talk is brief and focused on the serving pattern rather than on operational details of a complete production system.

## Key ideas
### Fine-tuning can make models cheaper to serve
[01:36](https://www.youtube.com/watch?v=O-UUy0ZiaBI&t=96s)
Dzhulgakov says fine-tuning gives businesses more control over outputs than adding a few examples to a prompt. It can also reduce serving cost because shorter prompts are processed faster and more cheaply. A smaller model may handle a narrow task after fine-tuning, where a much larger model would otherwise be needed. He gives the example of using a 7-billion-parameter model for a specialized task instead of relying on a 70-billion-parameter model.

### A shared base model lets many engineers run separate experiments
[02:41](https://www.youtube.com/watch?v=O-UUy0ZiaBI&t=161s)
With foundation models, one base model may support many tasks. Dzhulgakov describes a pattern in which engineers branch from that model and create separate fine-tuned variants for production tasks or experiments. They can test these variants in parallel without changing one another's work. Teams can later combine their best discoveries, merge datasets, or perform a larger training run on the base model.

### Parameter-efficient fine-tuning reduces the training burden
[04:17](https://www.youtube.com/watch?v=O-UUy0ZiaBI&t=257s)
Full fine-tuning can be expensive and difficult to get right on small datasets. Dzhulgakov describes parameter-efficient fine-tuning, where the main model remains in inference mode and only a smaller set of parameters is learned. He names LoRA as one widely used method. LoRA adds a small adapter to each model layer, reducing the number of parameters trained and the GPU resources needed.

### Many lightly used variants create a GPU utilization problem
[05:21](https://www.youtube.com/watch?v=O-UUy0ZiaBI&t=321s)
LLM inference is heavily constrained by moving data from memory. Serving one model efficiently therefore depends on sending many requests to it at the same time and batching them. Dzhulgakov says that parallel requests can improve efficiency substantially. The same approach breaks down when a team has dozens or hundreds of fine-tuned variants, because each variant may need separate GPU allocation while receiving little traffic.

### Shared adapters allow related variants to use the same GPU
[07:04](https://www.youtube.com/watch?v=O-UUy0ZiaBI&t=424s)
Fireworks.ai groups variants that were fine-tuned from the same base model and deploys them on shared hardware. Requests for those variants can be processed together even though they use different LoRA adapters. This preserves batching efficiency while allowing teams to customize the model. A newly fine-tuned variant can also be uploaded and deployed on existing hardware instead of requiring a new allocation.

### The platform applies inference optimizations to reduce per-token cost
[08:26](https://www.youtube.com/watch?v=O-UUy0ZiaBI&t=506s)
Dzhulgakov says Fireworks.ai combines cross-model batching with other LLM inference techniques. He mentions multi-attention, different forms of batching, disaggregating compute, and hardware configurations for particular use cases. The stated goal is to reduce the end-to-end cost of deploying and serving fine-tuned and base models compared with platforms that charge for separately allocated GPU hours.

### Fireworks.ai supports existing OpenAI-style integrations
[09:15](https://www.youtube.com/watch?v=O-UUy0ZiaBI&t=555s)
The platform offers open-source models through its inference engine and is compatible with the OpenAI API format. Dzhulgakov says developers can use existing integrations or change the base URL in an existing SDK. He also mentions a LangChain integration and access through the Fireworks.ai REST API.

## Notable quotes
- "Fine-tuning gives you more control over the output." (01:36)
- "If you have low-rank adapters, it allows you to also serve it more efficiently at runtime for deployment." (04:59)
- "You need to send a lot of requests to a single model in parallel and kind of batch them together for parallel processing." (06:26)
- "You can kind of be clever and deploy them on the same GPU." (07:04)
- "We are OpenAI compatible, which means that you can use existing integrations or just change the base URL in an existing SDK." (09:45)

## Tools & references mentioned
- Fireworks.ai
- PyTorch
- Meta
- OpenAI
- GPT-3.5 Turbo
- LoRA
- LangChain

## Who should watch
- You are testing several fine-tuned versions of an open-source model and are paying for separate GPU allocations that stay mostly idle.
- Your team needs a practical explanation of why batching works for one busy model but becomes harder with many lightly used variants.
- You want to connect an OpenAI-style client or LangChain application to an inference service for open-source models.

## Related talks

- [Scalable Evaluation and Serving of Open Source LLMs](https://mlopstalks.com/talks/scalable-evaluation-and-serving-of-open-source-llms) (Waleed Kadous, Anyscale, 34:57)
- [Cost Optimization and Performance](https://mlopstalks.com/talks/cost-optimization-and-performance) (Lina Weichbrodt & Luis Ceze, OctoML & Jared Zoneraich, Prompt Layer & Daniel Campos, Neeva & Mario Kostelac, Intercom, 36:06)
- [How We Cut LLM Latency 70% With TensorRT in Production](https://mlopstalks.com/talks/how-we-cut-llm-latency-70-with-tensorrt-in-production) (Maher Hanafi, Betterworks, 1:05:20)
- [Considerations and Optimizations for Deploying Open Source LLMs at Your Company](https://mlopstalks.com/talks/considerations-and-optimizations-for-deploying-open-source-llms-at-your-company) (Oscar Rovira, Mystic AI, 11:31)
- [Fine Tuning Llamas](https://mlopstalks.com/talks/fine-tuning-llamas) (Kai Davenport, 13:37)