# Streamlining Model Deployment

Daniel Lenton, Unify | AI in Production 2024 | 21:39

Source: https://www.youtube.com/watch?v=jodNnvBFYws
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/streamlining-model-deployment
Published: 2024-05-01
Tags: benchmarks, cost, latency, model-serving

## TL;DR
- The large number of models, endpoints, runtimes, compilers, and hardware choices creates a large search space for deployment decisions.
- Runtime benchmarks should use open governance, a standard tokenizer, varied inputs, and time-series measurements that reflect what users observe.
- Dynamic routing can select the best provider for a chosen metric, such as time to first token, token throughput, or cost, without changing the client API.

## Summary
Daniel Lenton argues that deploying AI models has become difficult because the number of endpoints, models, serving systems, compression methods, compilers, and hardware options keeps growing. Each application has a different trade-off between cost, latency, throughput, memory, and quality. Static leaderboards do not describe runtime performance well because endpoint systems change with traffic, geography, software updates, and hardware updates. Lenton proposes open, target-agnostic runtime benchmarks with a standard tokenizer, varied inputs, different concurrency and geographic conditions, and user-facing metrics such as time to first token and inter-token latency. Unify's model hub uses these measurements to route requests dynamically. A client can ask for the provider with the lowest first-token latency, highest token rate, or lowest cost. Lenton also discusses future quality-based routing, where a request could go to a smaller model when it is sufficient or to a specialist model for tasks such as text-to-SQL. He is clear that quality benchmarking remains harder than runtime measurement because outputs are noisy.

## Key ideas
### The endpoint market has become a large deployment search space
[01:01](https://www.youtube.com/watch?v=jodNnvBFYws&t=61s)
Lenton describes a rapid increase in AI-as-a-service endpoints, including existing models exposed through efficient APIs and specialist systems such as text-to-image and text-to-SQL models. The choices extend beyond endpoints. Teams must also consider new model architectures, serverless GPU options, orchestration systems, compression methods, compilers, and hardware from vendors such as Nvidia, Intel, AMD, and others. These layers combine into a large search space, with many ways to change the performance of a deployed model.

### Deployment decisions require trade-offs between several performance goals
[03:04](https://www.youtube.com/watch?v=jodNnvBFYws&t=184s)
Lenton says applications may optimize for low cost, low latency, high accuracy, high throughput, or low memory use. These goals can conflict. More idle devices can increase throughput while raising cost. Heavy compression can reduce latency while sacrificing accuracy. Because every application has its own preferred combination, selecting an endpoint is difficult when relying only on the options currently visible to the user.

### Runtime performance should be measured as a changing system
[04:19](https://www.youtube.com/watch?v=jodNnvBFYws&t=259s)
Endpoint performance changes with load balancing, user traffic, software and hardware updates, network speed, and geography. Lenton argues that runtime performance is a property of a dynamic system rather than a fixed property of model weights. He proposes open-source, target-agnostic benchmarks governed openly and built around metrics that users directly experience, including time to first token and inter-token latency.

### Benchmark comparisons need consistent inputs and measurement conditions
[05:24](https://www.youtube.com/watch?v=jodNnvBFYws&t=324s)
Lenton says runtime comparisons need a uniform tokenizer because tokenization changes token counts, tokens-per-second measurements, and token costs. Inputs should vary because repetitive prompts can affect techniques such as speculative decoding. Tests should also cover different concurrency levels and geographic locations. These conditions help describe how an endpoint behaves in practice instead of producing one isolated score.

### Time-series dashboards reveal why static provider selection is inefficient
[07:00](https://www.youtube.com/watch?v=jodNnvBFYws&t=420s)
Unify's model hub provides runtime dashboards that track endpoint metrics over time. Lenton describes a period in which different providers take the top position at different points. This leads him to treat runtime benchmarking as a time-series problem. Sending every request to one provider means accepting its slow periods, while a router can select the provider currently performing best for the chosen metric.

### Dynamic routing can optimize a request without changing the application interface
[08:01](https://www.youtube.com/watch?v=jodNnvBFYws&t=481s)
The routing approach takes the best current provider across a group of endpoints. Users can optimize for time to first token, inter-token latency, throughput, input cost, or output cost. In the demo, one agent selects a provider for first-token speed and another selects one for token rate. The same interface can also choose the cheapest provider for input-heavy chat or output-heavy content generation tasks.

### Quality-based routing is promising but harder to measure
[15:27](https://www.youtube.com/watch?v=jodNnvBFYws&t=927s)
Lenton says the next step is routing based on the quality needed for an individual prompt. A simple question such as the capital of Egypt may not need a large model, while a text-to-SQL request may benefit from a specialist model. Human evaluations or language-model evaluators could help train this routing system. He also discusses routing across modalities. During questions, he acknowledges that provider-specific compression, randomness, or other changes can make the same nominal model produce different quality, so quality needs its own time-series benchmarks.

## Notable quotes
- "The combinatorial search space across all of these is very vast." (02:46)
- "We need to think about them as dynamic systems rather than kind of static leaderboards." (04:49)
- "You can get performance which is better than any individual provider." (08:36)
- "Runtime is relatively objective, so there's not too much disputing. Quality is a whole other game altogether." (18:59)

## Tools & references mentioned
- Unify
- AI-as-a-service
- ChatGPT
- AnyScale
- Ray
- OctoAI
- Together AI
- Stable Diffusion
- GPT-4
- Gemini
- Gemma
- Mixtral
- Mamba
- XLA
- TensorRT
- OpenAI Triton
- Mojo
- Nvidia
- Intel
- AMD
- SambaNova
- Cerebras
- Llama 2 70B Chat
- Code Llama 34B Instruct
- Mistral
- Chatbot Arena
- LM Per
- Fireworks
- Martian

## Who should watch
- You are choosing between multiple model providers and want routing based on current latency, throughput, or cost rather than a fixed endpoint.
- Your team needs runtime measurements that account for traffic, geography, concurrency, and changing provider infrastructure.
- You are building model selection systems and want to understand why prompt-level quality routing is harder than runtime routing.

## Related talks

- [Efficiently Scaling and Deploying LLMs](https://mlopstalks.com/talks/efficiently-scaling-and-deploying-llms) (Hanlin Tang, MosaicML, 25:14)
- [Different Ways of Serving ML Models](https://mlopstalks.com/talks/different-ways-of-serving-ml-models) (Byron Allen, 1:04:46)
- [Scalable Evaluation and Serving of Open Source LLMs](https://mlopstalks.com/talks/scalable-evaluation-and-serving-of-open-source-llms) (Waleed Kadous, Anyscale, 34:57)
- [DSPy: Transforming Language Model Calls into Smart Pipelines](https://mlopstalks.com/talks/dspy-transforming-language-model-calls-into-smart-pipelines) (Omar Khattab, Stanford, 1:05:40)
- [Considerations and Optimizations for Deploying Open Source LLMs at Your Company](https://mlopstalks.com/talks/considerations-and-optimizations-for-deploying-open-source-llms-at-your-company) (Oscar Rovira, Mystic AI, 11:31)