# Scalable Evaluation and Serving of Open Source LLMs

Waleed Kadous, Anyscale | LLMs in Production 2023 | 34:57

Source: https://www.youtube.com/watch?v=xQRdOkVCAUY
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/scalable-evaluation-and-serving-of-open-source-llms
Published: 2023-07-03
Tags: evals, gpus, model-serving, open-models

## TL;DR
- Self-hosted LLMs give teams more control over data, costs, and model customization, although commercial models can still provide better quality for some tasks.
- Choosing an open source model requires comparing output quality, latency, token cost, model size, fine-tuning, and the way each model uses prompts and stop tokens.
- Production serving needs infrastructure for multiple models, GPU memory, batching, fault tolerance, autoscaling, and fast model startup.

## Summary
Waleed Kadous shares the first production lessons from building and running Aviary, Anyscale's open source LLM serving and evaluation system. He compares commercial APIs, hosted open source services, and self-hosted models, arguing that self-hosting can make sense when data control, cost, or customization matters. He demonstrates Aviary Explorer, which compares models using prompts, output quality, latency, and token cost, along with command-line tools for batch evaluation and GPT-4-based answer comparison. The harder part is serving the models. Different models need different prompt formats, stop tokens, accelerators, GPU counts, batching, and streaming settings. Teams also tend to deploy more models than expected, so declarative model configurations and a router become useful. Kadous explains why GPU memory matters more than raw compute for batching, and why caching, fast startup, and carefully tuned autoscaling are needed when traffic is spiky. Aviary is open source apart from Anyscale's optimized node startup service.

## Key ideas
### Self-hosting makes sense when data control, cost, or customization matters
[05:16](https://www.youtube.com/watch?v=xQRdOkVCAUY&t=316s)
Waleed frames self-hosting as a choice that depends on the job rather than a universal replacement for commercial APIs. Teams may need to keep data under their control because of regulation or competitive concerns. Cost is another reason. He gives the example of a document search service backed by GPT-4 that costs 25 cents per query, and says GPT-4 can cost 10 cents per thousand tokens at the high end. Self-hosting also allows teams to fine-tune models without paying the much higher serving price that commercial providers may charge. He is clear about the tradeoff: open source quality still trails the best commercial models, although he says it can already be sufficient for summarization, retrieval, and assistant generation.

### Model selection needs direct comparison of quality, latency, and cost
[08:05](https://www.youtube.com/watch?v=xQRdOkVCAUY&t=485s)
Aviary Explorer lets users select models, send the same prompt to several of them, and compare their outputs. The interface records latency, cost, and token use, while users can vote for the answer they prefer. Those votes can support a use-case-specific leaderboard rather than a generic ranking. Waleed warns that the displayed figures simplify batching and should be treated as an upper bound, since batching can have a 10x effect. He also points out that a model can appear expensive because it generates long answers, even when its cost per token is not unusually high. The command-line tools extend this process to prompt files and comparisons with OpenAI.

### Every model needs configuration for its own prompt and serving behavior
[13:42](https://www.youtube.com/watch?v=xQRdOkVCAUY&t=822s)
Downloading a model from Hugging Face and running it is not enough for a production service. Waleed says models differ in role definitions, marker strings, stop tokens, accelerator support, GPU requirements, batching, and streaming. Aviary handles these differences through a model configuration file for each model. The files specify the strings and settings needed to produce sensible output. This abstraction also makes it possible to add a new model quickly or run a fine-tuned version by changing the configuration. He demonstrates that the configuration approach allowed the team to add support for a newly released model in about five minutes.

### Production systems usually need more models than the original plan
[18:00](https://www.youtube.com/watch?v=xQRdOkVCAUY&t=1080s)
The team initially expected to run a small, medium, and large model. That quickly expanded because different model sizes, fine-tunes, and A/B tests created separate deployment needs. Waleed also describes a router pattern in which specialized models handle different tasks and another model decides which one to call. Even a simple version of that design requires several models. Aviary therefore treats model deployments as a group rather than as individually managed pets. Its backend has a deployment for each model and a router deployment that directs traffic. Ray and Ray Serve can replace a failed server, which lets the system manage a changing collection of models more uniformly.

### GPU memory determines how much useful batching a model can support
[23:03](https://www.youtube.com/watch?v=xQRdOkVCAUY&t=1383s)
Waleed says LLM serving uses GPU memory for model parameters and for working memory used during batching. Batching sends several requests to the GPU together and can greatly increase throughput, with a small latency cost. This makes memory capacity more important than a small difference in raw GPU speed. He says the team moved away from 16 GB GPUs and generally preferred larger-memory instances. Very large models such as Falcon and Open Assistant with 30 or 40 billion parameters may require two A100 GPUs because the parameters do not fit on one. The practical decision is to leave enough memory for batching, rather than selecting a GPU based only on compute speed.

### Spiky traffic makes startup time and autoscaling settings part of serving design
[25:25](https://www.youtube.com/watch?v=xQRdOkVCAUY&t=1525s)
Aviary keeps at least one replica of each model running because scaling from zero can add a minute or two when a user first requests a model. With 12 models, that means keeping 12 GPUs available after optimization. Autoscaling alone is not enough if a peak lasts only 20 minutes and the system needs 15 minutes to react. The team caches Hugging Face models locally and uses fast local storage to reduce node startup time. Waleed says Anyscale's proprietary service can start nodes in under a minute. The autoscaler also needs carefully chosen minimum replicas, lookback intervals, and slower downscaling, because traffic can change sharply and a recent article caused the site's traffic to quadruple.

### Token rates may be better autoscaling signals than request counts
[32:23](https://www.youtube.com/watch?v=xQRdOkVCAUY&t=1943s)
In the question period, Waleed says the team is still tuning the metric used for autoscaling. LLM requests vary widely because a short prompt can produce a long answer, while another request may produce only a few words. Request counts or memory use alone can therefore give a misleading view of load. He says continuous batching and streaming help even out that variation. Measuring the rate of tokens coming in and the rate of tokens going out can provide a more accurate basis for predicting demand. He does not present this as a finished solution, since the team is still working on the tuning.

## Notable quotes
- "You just want to deploy the right tool for the job." (05:13)
- "Deploying LLMs is harder than it looks." (13:42)
- "It's all about the GPU RAM, not about compute." (23:03)
- "The thing we're optimizing for is making sure there's enough GPU memory that we can do batching." (24:25)
- "If you can, do it. What we found is the peaks are very, very random." (32:00)

## Tools & references mentioned
- Anyscale
- Ray
- Ray Serve
- Aviary
- Aviary Explorer
- OpenAI
- Anthropic
- Hugging Face
- MosaicML
- OctoAI
- Falcon
- GPT-4
- Cohere
- Kappa
- Open Assistant
- StoryWriter
- LM Systems
- Amazon
- MPT
- S3
- A100
- llmnumbers.ray.io

## Who should watch
- You are deciding whether to use a commercial API, a hosted open source model, or your own serving stack.
- You need to compare several open source models on your own prompts instead of relying on a general leaderboard.
- You are preparing to serve multiple LLMs and need practical guidance on GPU memory, batching, startup time, and autoscaling.

## Related talks

- [Considerations and Optimizations for Deploying Open Source LLMs at Your Company](https://mlopstalks.com/talks/considerations-and-optimizations-for-deploying-open-source-llms-at-your-company) (Oscar Rovira, Mystic AI, 11:31)
- [Challenges in Providing LLMs as a Service](https://mlopstalks.com/talks/challenges-in-providing-llms-as-a-service) (Hemant Jain, Cohere AI, 11:43)
- [The Truth About LLM Training](https://mlopstalks.com/talks/the-truth-about-llm-training) (Paul van der Boor & Zulkuf Genc, Prosus Group, 55:47)
- [Graduating from Proprietary to Open Source Models in Production](https://mlopstalks.com/talks/graduating-from-proprietary-to-open-source-models-in-production) (Philip Kiely, Baseten, 23:16)
- [LLMs For the Rest of Us](https://mlopstalks.com/talks/llms-for-the-rest-of-us) (Vikram Sreekanti, Aqueduct & Joseph Gonzalez, UC Berkeley and Aqueduct, 24:33)