# Build and Customize LLMs in Less than 10 Lines of YAML

Travis Addair, Predibase | LLMs in Production 2023 | 34:05

Source: https://www.youtube.com/watch?v=De6RY2GN-e4
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/build-and-customize-llms-in-less-than-10-lines-of-yaml
Published: 2023-07-05
Tags: fine-tuning, model-serving, rag, structured-outputs

## TL;DR
- Travis Addair argues that most teams should customize existing foundation models instead of training large language models from scratch.
- Ludwig lets users move between zero-shot prompting, retrieval, and fine-tuning by changing a small declarative YAML configuration.
- Fine-tuning quality depends more on having the right data and model head for the task than on adjusting many training parameters.

## Summary
Travis Addair explains how teams can adapt existing large language models to specific business tasks without taking on the cost of pretraining. He describes a progression from zero-shot prompting to retrieval with indexed examples, then to fine-tuning when a labeled data set and stronger task performance justify it. Ludwig provides the declarative configuration layer, while Predibase manages data connections, model versions, training, deployment, and querying. The examples use hotel reviews, including retrieving reviews about poor Wi-Fi and predicting ratings from one to five. Addair also explains why classification tasks often need a classification head rather than text generation. He recommends starting with frozen model weights, smaller models, and parameter-efficient methods such as LoRA. In the demo, semantic retrieval improves the example accuracy from 37 to 66. He is direct about the trade-offs: fine-tuning can reduce latency and cost, but the data and task formulation matter more than endlessly tuning parameters.

## Key ideas
### LLM applications begin with an existing foundation model
[06:42](https://www.youtube.com/watch?v=De6RY2GN-e4&t=402s)
Addair contrasts the traditional machine learning lifecycle with the newer LLM workflow. Traditional systems require data preparation, feature engineering, training, evaluation, and deployment before they produce business value. Foundation models have already absorbed much of the general-purpose training cost. Teams can start from a model in a registry such as the Hugging Face Hub, then apply prompting, retrieval, or fine-tuning. He says most organizations do not need to build general intelligence because they need a model that performs a specific organizational task.

### Teams can move from prompting to retrieval and fine-tuning as their needs grow
[08:28](https://www.youtube.com/watch?v=De6RY2GN-e4&t=508s)
The proposed workflow starts with the least expensive intervention. A team can prompt a model, add indexed examples for retrieval-augmented or few-shot inference, and later build a labeled data set from feedback and annotations. Fine-tuning becomes useful when the team needs more consistent behavior, lower latency, or higher throughput. This progression avoids paying the full cost of training at the beginning and lets teams learn about the task and data before committing to a more involved approach.

### Declarative configuration hides much of the infrastructure needed for LLM customization
[10:53](https://www.youtube.com/watch?v=De6RY2GN-e4&t=653s)
Addair describes Ludwig as a declarative framework similar in spirit to SQL. Users describe their inputs, outputs, task, retrieval strategy, and training settings, while the underlying system turns that description into training or inference work. A zero-shot configuration can use a Llama model without training. Additional configuration enables semantic retrieval and selects the top three examples for a prompt. A training section then adds supervised fine-tuning parameters such as the optimizer and learning rate.

### Predibase adds managed data, model, infrastructure, and deployment workflows
[12:47](https://www.youtube.com/watch?v=De6RY2GN-e4&t=767s)
Predibase layers a managed platform on top of Ludwig. Addair names connectors for Snowflake, Databricks, S3, and other data stores, along with a model repository for experiments and trained weights. Users can query models interactively, run batch predictions, and use indexed data from a UI, Python SDK, or command-line tool. The platform also provides managed training and prediction infrastructure, including resource selection and distributed training support.

### The right fine-tuning architecture depends on the task output
[17:38](https://www.youtube.com/watch?v=De6RY2GN-e4&t=1058s)
For generative tasks, users can fine-tune the full text-to-text model. Classification tasks often need a different design. Addair describes removing the language-model head, taking the model's hidden state, and attaching a classification head with a fixed set of categories. Ludwig exposes these choices through configuration, including whether the base model weights remain frozen. Freezing the model usually reduces cost and training time, although training more weights can improve performance when it is needed.

### Freezing the base model and using optimizations can make fine-tuning much faster
[19:47](https://www.youtube.com/watch?v=De6RY2GN-e4&t=1187s)
Addair recommends beginning experiments without training the language-model weights. In his IMDb review example, full fine-tuning took about 60 minutes, while the optimized setup took less than a minute. The faster path used automatic mixed precision and cached encoder embeddings, then trained fully connected layers on top. He also describes LoRA as a parameter-efficient option and says Ludwig supports distributed training with model and data parallelism.

### Retrieval can solve a knowledge gap before fine-tuning is warranted
[22:42](https://www.youtube.com/watch?v=De6RY2GN-e4&t=1362s)
In the demo, the model can answer general questions such as naming popular hotel chains, but it does not know which hotels in the supplied data have poor Wi-Fi. Addair adds a data set to index rather than immediately fine-tuning the model. Predibase retrieves relevant hotel reviews and inserts them into the prompt. The response then cites examples from the provided context. He presents this as a practical way to inspect a data set and improve answers without changing model weights.

### Structured outputs make LLM predictions usable as supervised tasks
[24:21](https://www.youtube.com/watch?v=De6RY2GN-e4&t=1461s)
The hotel-rating example shows the model predicting a score from one to five. Initially, the model produces verbose answers, so Addair iterates on the prompt until the output is closer to a simple rating format. He then formalizes the task in the model repository by defining a category output vocabulary and validators. The demo compares zero-shot, random retrieval, and semantic retrieval. The displayed accuracy rises from 37 for zero-shot to 66 with semantic retrieval.

### Fine-tuning decisions should start with data quality and task fit
[29:37](https://www.youtube.com/watch?v=De6RY2GN-e4&t=1777s)
In the questions, Addair says the most important part of fine-tuning is putting the data in the right form. Parameter-efficient methods such as LoRA can reduce catastrophic forgetting because they adjust fewer weights. He also advises matching the technique to the task, using a classification head for classification instead of forcing the model to generate class names as text. Starting with a smaller model improves iteration speed and may perform as well as a much larger model when the task is bounded.

## Notable quotes
- "The value comes at the very end, once you have something live in production that you can actually start generating predictions from." (05:56)
- "What you really need is this solution-oriented flow here, which is all about getting the model to do something specific for your organization." (09:05)
- "The much more important thing is to make sure your data is in the right form." (29:51)
- "I always recommend starting when doing experiments with not training the weights of the LLM." (19:39)

## Tools & references mentioned
- Predibase
- Ludwig
- Uber
- Linux Foundation
- Horovod
- Michelangelo
- OpenAI
- Google
- Meta
- Hugging Face Hub
- Llama
- Falcon
- PyTorch
- DeepSpeed
- LoRA
- AdaLoRA
- Snowflake
- Databricks
- S3

## Who should watch
- You are choosing between prompting, retrieval, and fine-tuning for a task-specific LLM and want a practical progression.
- You need structured classification or prediction outputs from a language model and want to understand when a classification head is a better fit.
- You are evaluating a declarative or managed workflow for training, querying, and deploying customized open-source models.

## Related talks

- [Fine-Tuning LLMs: Best Practices and When to Go Small](https://mlopstalks.com/talks/fine-tuning-llms-best-practices-and-when-to-go-small) (Mark Kim-Huang, Preemo Inc., 53:48)
- [LLMs Mini Summit // MLOps Mini Summit Meetup #3](https://mlopstalks.com/talks/llms-mini-summit-mlops-mini-summit-meetup-3) (Thomas Capelle, Weights & Biases & Boris Dayma, Craiyon & Jonathan Whitaker & Robbie McCorkell, Leap Labs, 1:45:28)
- [Pitfalls and Best Practices: 5 Lessons from LLMs in Production](https://mlopstalks.com/talks/pitfalls-and-best-practices-5-lessons-from-llms-in-production) (Raza Habib, Humanloop, 30:27)
- [The Emerging Toolkit for Reliable, High-quality LLM Applications](https://mlopstalks.com/talks/the-emerging-toolkit-for-reliable-high-quality-llm-applications) (Matei Zaharia, Databricks, 31:01)
- [LangChain: Enabling LLMs to Use Tools](https://mlopstalks.com/talks/langchain-enabling-llms-to-use-tools) (Harrison Chase, LangChain, 11:43)