# End-to-end Modern Machine Learning in Production

Omar Sanseviero, Hugging Face | LLMs in Production 2023 | 10:30

Source: https://www.youtube.com/watch?v=wH4Gqrv_v40
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/end-to-end-modern-machine-learning-in-production
Published: 2023-08-11
Tags: fine-tuning, inference, model-serving, open-source

## TL;DR
- Modern language models are difficult to run because they are large, have high memory requirements, and need evaluation against real use cases.
- Quantization, tensor parallelism, token streaming, and Text Generation Inference reduce the practical cost and latency of serving language models.
- Parameter-efficient fine-tuning trains small adapters on top of a frozen base model, making adaptation possible with much less memory and compute.

## Summary
Omar Sanseviero gives a short tour of tools and techniques for putting large language models into products. He first covers inference, using examples such as StarCoder, Llama, and Falcon. Model size creates immediate infrastructure problems: Falcon can require 90 gigabytes of GPU memory, which exceeds a single A100. Quantization can reduce that requirement, while tensor parallelism spreads a model across GPUs. Text Generation Inference adds serving features such as token streaming, monitoring, quantization support, and Flash Attention. Omar then moves to adapting models for particular data and tasks. Parameter-efficient fine-tuning freezes the base model and trains small adapters instead. LoRA can make Whisper fine-tuning practical in Google Colab, and QLoRA combines 4-bit quantization with adapter tuning. The examples are early ecosystem tools, and Omar is careful to frame the talk as a high-level overview rather than a complete production guide.

## Key ideas
### Large language models create infrastructure problems before serving begins
[01:21](https://www.youtube.com/watch?v=wH4Gqrv_v40&t=81s)
Omar uses StarCoder, Llama, and Falcon to show how quickly the open model ecosystem was changing. These models range from several billion parameters to 65 billion parameters. Falcon can require 90 gigabytes of GPU memory, so it cannot fit on a single A100. Model size is only one issue. Standard benchmarks may not match real usage, companies often need to tune models for their own data, and users expect low latency.

### Quantization lowers the memory needed to load large models
[03:17](https://www.youtube.com/watch?v=wH4Gqrv_v40&t=197s)
Omar describes loading models in 8-bit or 4-bit modes with open source tools such as bitsandbytes and Accelerate. A 4-bit configuration can load the larger Falcon model with 27 gigabytes of RAM instead of the 90 gigabytes he cited for the full model. Some computation can also run on the CPU. That makes larger models usable with less hardware, although CPU offloading makes inference much slower.

### Text Generation Inference combines several serving optimizations
[03:59](https://www.youtube.com/watch?v=wH4Gqrv_v40&t=239s)
Text Generation Inference is an open source library for serving language models. Its tensor parallelism splits a tensor into slices that different GPUs process, allowing one model to use multiple GPUs. Token streaming sends generated tokens as soon as they are available instead of waiting for the complete response. Omar also mentions monitoring, quantization through bitsandbytes, and Flash Attention. He says Code in Chat, Open Assistant, and the LLM Playground use the library.

### Real-world evaluation matters more than leaderboard scores alone
[02:42](https://www.youtube.com/watch?v=wH4Gqrv_v40&t=162s)
Omar says language model benchmarks are not necessarily representative of the real situations where companies will use the models. Teams usually need to adjust a model for their own data and use case. They also need fast latency. This means selecting a model cannot stop at comparing public benchmark results. The intended product behavior and serving constraints have to shape the evaluation.

### Fine-tuning reuses an expensive base model with less data and compute
[05:35](https://www.youtube.com/watch?v=wH4Gqrv_v40&t=335s)
Traditional training needs substantial data and compute. Fine-tuning starts from a large model that a research lab has already trained, then adapts it to a company or personal dataset. Omar says this requires much less data and compute and can be completed faster. With very large language models, however, updating every parameter becomes difficult, which motivates parameter-efficient fine-tuning.

### Adapters make parameter-efficient fine-tuning possible on modest hardware
[06:52](https://www.youtube.com/watch?v=wH4Gqrv_v40&t=412s)
Parameter-efficient fine-tuning freezes the original model, adds a small adapter, and trains the adapter rather than the full model. Omar expects some quality loss, but says performance can remain nearly comparable while training becomes much faster. He uses LoRA with Whisper as an example. Full Whisper fine-tuning can run out of memory in Google Colab, while a LoRA adapter is much smaller and makes the task practical.

### QLoRA combines 4-bit quantization with adapter training
[08:33](https://www.youtube.com/watch?v=wH4Gqrv_v40&t=513s)
Omar describes QLoRA as a recent technique that fine-tunes a model with 4-bit quantization and adapter tuning. The combination reduces the memory needed for training. He connects this to reinforcement learning from human feedback, where one frozen base model can support multiple adapters, such as separate adapters for the preference model and other components.

## Notable quotes
- "My goal today is really to increase awareness of which are the existing tools out there that can make it easier for you to use the state of the art ML models in your products, in your services." (00:47)
- "Falcon, for example, requires a lot of GPU memory. It requires 90 gigabytes of GPU memory." (02:22)
- "Rather than tuning or training the full model, you will freeze the model, you will add some adapter or some additional parameters, and those are the ones that you will train." (07:13)
- "Everything is in GitHub. Everything is open source, so feel free to check it out, feel free to give some Stars." (09:36)

## Tools & references mentioned
- StarCoder
- Llama
- Falcon
- bitsandbytes
- Accelerate
- Text Generation Inference
- Code in Chat
- Open Assistant
- LLM Playground
- Whisper
- LoRA
- Stable Diffusion
- QLoRA
- Google Colab
- Reinforcement learning from human feedback

## Who should watch
- You are choosing how to serve an open language model and need a quick explanation of memory reduction, multi-GPU serving, and token streaming.
- You want to fine-tune Whisper, Falcon, or another large model without paying for a large GPU setup.
- You are trying to understand where parameter-efficient fine-tuning fits into a reinforcement learning from human feedback workflow.

## Related talks

- [Graduating from Proprietary to Open Source Models in Production](https://mlopstalks.com/talks/graduating-from-proprietary-to-open-source-models-in-production) (Philip Kiely, Baseten, 23:16)
- [ML Scalability Challenges](https://mlopstalks.com/talks/ml-scalability-challenges) (Waleed Kadous, Anyscale, 1:00:03)
- [Fine-Tuning LLMs: Best Practices and When to Go Small](https://mlopstalks.com/talks/fine-tuning-llms-best-practices-and-when-to-go-small) (Mark Kim-Huang, Preemo Inc., 53:48)
- [Let's Continue Bundling into the Database](https://mlopstalks.com/talks/lets-continue-bundling-into-the-database) (Ethan Rosenthal, Square, 51:56)
- [Efficiently Scaling and Deploying LLMs](https://mlopstalks.com/talks/efficiently-scaling-and-deploying-llms) (Hanlin Tang, MosaicML, 25:14)