# LLMs For the Rest of Us

Vikram Sreekanti, Aqueduct & Joseph Gonzalez, UC Berkeley and Aqueduct | LLMs in Production 2023 | 24:33

Source: https://www.youtube.com/watch?v=AaM6nF1a784
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/llms-for-the-rest-of-us
Published: 2023-07-27
Tags: evals, model-serving, open-models, rag

## TL;DR
- Open LLMs improved quickly because better training data and instruction tuning produced strong results with relatively small models.
- The main difficulty in turning an LLM into a product is the surrounding software, data, infrastructure, monitoring, governance, and cost management.
- Teams need infrastructure that can accommodate rapidly changing models and tools instead of locking them into one opinionated stack.

## Summary
Joseph Gonzalez describes two perspectives on LLMs: Berkeley's work on model and systems research, and Aqueduct's focus on what software teams need today. He explains how Llama, Alpaca, ShareGPT data, and the Vicuna project helped open models improve quickly. The Chatbot Arena then provided a way to compare models through human preferences and pairwise battles. Gonzalez argues that innovation will increasingly concern how models are used inside applications, including tools such as Gorilla for launching cloud resources from natural-language instructions. Vikram Sreekanti then describes the missing software around an LLM application. Teams need model access, retrieval, prompt chains, ordinary Python code, cloud deployment, data controls, telemetry, and cost management. Their advice is to start with a model that is easy to try, use retrieval rather than stuffing prompts with data, and choose infrastructure that can keep changing as the ecosystem develops.

## Key ideas
### Research and production are moving closer together
[02:08](https://www.youtube.com/watch?v=AaM6nF1a784&t=128s)
Joseph Gonzalez says Berkeley works on frontier problems in model architectures, evaluation, inference, training, and memory use, while Aqueduct studies what teams can actually build now. He says these paths are colliding for the first time because research advances are affecting products within weeks rather than years. Aqueduct's goal is to help every software team build LLM-powered applications. The talk combines the research view with the practical problems that appear when teams try to deploy these systems.

### Better data made small open models much more capable
[04:49](https://www.youtube.com/watch?v=AaM6nF1a784&t=289s)
Gonzalez traces the open-model progress from Llama to Alpaca and Vicuna. Llama used better training data and training methods, although it was weak at following instructions. Alpaca fine-tuned Llama with self-instruction data to make it behave more like a conversational assistant. The Vicuna team used about 70,000 shared ChatGPT conversations from ShareGPT, around 800 megabytes of high-quality, multi-turn examples. Gonzalez says the first Vicuna training run was done by a few students in about a week and cost a few hundred dollars.

### Model evaluation needs open-ended and human-based tests
[07:05](https://www.youtube.com/watch?v=AaM6nF1a784&t=425s)
The team built a GPT-4 evaluation method because standard NLP benchmarks did not capture instruction following well. GPT-4 judged answers on criteria such as engagement, insightfulness, and factuality. They later created Chatbot Arena, where people ask randomly paired models questions and choose between the responses. The results are scored with the Elo rating system. Gonzalez also describes limitations: some models abstain on unusual questions, while humans may prefer any answer, so a richer benchmark is needed.

### Application design will matter more than training another model
[10:37](https://www.youtube.com/watch?v=AaM6nF1a784&t=637s)
Gonzalez says the field may be shifting from building models to using them in applications. His group built Gorilla to help researchers obtain cloud resources through natural-language instructions, such as launching virtual machines with particular GPUs in a specified region. The model reads documentation and uses it to invoke cloud services. He presents this as an example of an LLM making an existing workflow easier rather than as a new general-purpose model.

### Start with an accessible model and retrieve the right data
[14:29](https://www.youtube.com/watch?v=AaM6nF1a784&t=869s)
Sreekanti advises teams to begin with the largest model they can easily use, which will often be a hosted model. Sensitive industries may later need another arrangement, but a hosted API can help prove that an application works. He warns against relying on a clever prompt with no data and against putting all available data into every prompt. Vector databases can retrieve relevant information at the right time and provide a more focused context.

### An LLM application still needs ordinary software around the model
[16:59](https://www.youtube.com/watch?v=AaM6nF1a784&t=1019s)
Sreekanti says models cannot handle every part of an application. Teams still need code to retrieve data from APIs, clean and featurize it, call the model, validate the result, and connect the workflow to the rest of the product. Tools such as LangChain and LlamaIndex help teams chain prompts, compare models, and experiment. The model is one component inside a larger application, not the entire application.

### Deployment creates problems that prototypes hide
[17:50](https://www.youtube.com/watch?v=AaM6nF1a784&t=1070s)
After a prototype works, teams must connect it to cloud infrastructure and the existing product. They need controls around data access so that information is not exposed accidentally, along with governance for the systems that feed the application. They also need records of inputs and outputs for debugging and complaints. Sreekanti adds budgeting and cost management because hosted APIs and self-managed open models can produce large bills quickly.

### Fast-changing tools require infrastructure that stays flexible
[19:34](https://www.youtube.com/watch?v=AaM6nF1a784&t=1174s)
Sreekanti says innovation is arriving at every layer, including models, databases, and Python libraries. Teams are adopting new research outputs quickly, sometimes only days after release. He expects teams to choose infrastructure that helps them keep up with these changes. In his view, infrastructure cannot be built around supporting one preferred tool while rejecting others, because the surrounding ecosystem will continue to change.

### Retrieval can help with latency and context limits
[21:59](https://www.youtube.com/watch?v=AaM6nF1a784&t=1319s)
In the question session, Gonzalez mentions memory-bandwidth techniques, batching, hardware utilization, and quantization as ways to improve serving performance. For long context, he is cautious about simply increasing the context window. He recommends retrieval methods that select the information needed for a request, because too much context can distract the model. He also expects LLM tooling to spread beyond Python, with implementations for languages such as Go, Rust, JavaScript, and TypeScript.

## Notable quotes
- Joseph Gonzalez: "Our goal at Aqueduct is to enable every software team to be able to build LLM-powered applications." (03:13)
- Joseph Gonzalez: "It's really all about the data." (06:01)
- Vikram Sreekanti: "The place to start is probably by picking a hosted model." (14:42)
- Vikram Sreekanti: "At the end of the day, to use these models, just like with any machine learning model, you're going to want to take some code that retrieves some data from an API, cleans it, featurizes it, passes it into the model, takes the result, validates it." (17:09)
- Joseph Gonzalez: "If you stuff too much in the context, that's often actually a bad sign." (22:51)

## Tools & references mentioned
- Aqueduct
- UC Berkeley RISE Lab
- Apache Spark
- ChatGPT
- Stable Diffusion
- Llama
- Alpaca
- ShareGPT
- Vicuna
- GPT-4
- GPT-3.5
- Claude
- PaLM 2
- Gorilla
- Chatbot Arena
- arena.lmsys.org
- LangChain
- LlamaIndex
- Python
- Go
- Rust
- JavaScript
- TypeScript

## Who should watch
- You are building an LLM prototype and need to understand what has to exist around the model before it can become a product.
- Your team is deciding whether to use hosted models, open models, retrieval, or prompt-chaining tools.
- You are responsible for cloud deployment, data access, debugging, or costs for an LLM application.

## Editor's note

Vikram Sreekanti says an LLM prototype becomes a production problem when teams need deployment, data controls, telemetry, debugging records, and cost management around the model. ZenML records each pipeline run's steps, inputs, outputs, and code version, so teams can trace an artifact or model back to the data and code that produced it.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [Fireside Chat - The Future of LLMs](https://mlopstalks.com/talks/fireside-chat-the-future-of-llms) (David Hershey, Unusual Ventures & Daniel Jeffries, AI Infrastructure Alliance, 36:07)
- [LLMs in Production Conference - Part II](https://mlopstalks.com/talks/llms-in-production-conference-part-ii) (Matei Zaharia, Databricks and Stanford & Chip Huyen, 37:21)
- [Challenges in Providing LLMs as a Service](https://mlopstalks.com/talks/challenges-in-providing-llms-as-a-service) (Hemant Jain, Cohere AI, 11:43)
- [Real LLM Success Stories: How They Actually Work](https://mlopstalks.com/talks/real-llm-success-stories-how-they-actually-work) (Alex Strick van Linschoten, ZenML, 48:20)
- [The Next Revolution in AI: LLMs and Beyond](https://mlopstalks.com/talks/the-next-revolution-in-ai-llms-and-beyond) (, 13:47)