Custom language models give companies control over intellectual property, customer SLAs, domain behavior, and data location.
2
Synthetic data can increase dataset size while self-consistency and diversity checks help remove weak or repetitive examples.
3
LoRA reduces the memory needed for fine-tuning large models by freezing the original weights and learning a smaller set of parameters.
Summary
Mark Huang presents a practical workshop on adapting open language models for specialized applications. He compares closed models, open models, and smaller task-specific models, then argues that companies may need custom models for ownership, reliability against provider outages, domain expertise, privacy, and security. The workshop focuses on supervised instruction fine-tuning. Mark walks through a pipeline that generates prompts, produces model responses, filters them with self-consistency, and evaluates response diversity with ROUGE-L. He then shows why full fine-tuning is difficult: model weights, gradients, optimizer state, and activations quickly consume GPU memory. The practical solution is parameter-efficient fine-tuning with LoRA, which freezes the base model and trains low-rank updates. The code uses Flan-UL2 and synthetic GSM8K-style math data. Mark is candid that the work still requires expensive GPUs and difficult infrastructure, which motivates Preemo's managed fine-tuning and deployment platform.
Custom models address limits that closed APIs cannot remove
Mark frames the workshop around building specialized models instead of relying only on closed APIs. GPT-4 can be used for prompting, but it cannot be customized in the way an enterprise may need. He gives four reasons to own a model: keeping intellectual property inside the modeling pipeline, meeting customer SLAs when a provider has an outage, adding domain expertise, and keeping data inside a virtual private cloud or on-premises environment. Open models also let teams choose task-specific behavior rather than accepting the behavior of a general hosted model.
The task should determine the data and model strategy
Before fine-tuning, Mark says teams should select the task and identify suitable proxy datasets. He separates knowledge tasks from reasoning tasks. Knowledge tasks include named entity recognition, basic question answering, and work expected from a knowledge worker. Reasoning tasks include coding and mathematics, with coding split between generating code and explaining code. He points attendees to HELM as an open evaluation effort for comparing language models. Benchmark results can help teams decide whether a model already handles the desired task or whether custom training is justified.
Synthetic data can combine scale with a quality filter
Mark describes a data synthesis loop with a prompt generator, an inference model, a response post-processor, and an evaluator. The model generates several completions for a prompt. Self-consistency keeps the answer supported by the majority of completions, although Mark notes that this is not verification. An evaluator then decides whether the result should be retained or sent through another generation loop. The aim is to produce more examples without accepting every synthetic response. Mark later uses ROUGE-L scores to filter out responses that are too similar and preserve more diverse reasoning paths.
Instruction examples teach the model to expose its reasoning path
The notebook uses math question-and-answer pairs to create answers for questions that do not already have answers. Mark builds a prompt with eight exemplars and asks the model to solve each problem step by step. In his example, a question about three weekly sessions of three 60-meter sprints produces 540 meters and includes the calculation that leads to the answer. A second example asks for the total number of ants and bugs in a garden and produces 75 insects. The point is to provide rationales, so the model learns a reasoning pattern rather than only an answer token.
Sequence length and training state dominate GPU memory
Mark explains why full fine-tuning runs out of memory even with several large GPUs. A 15-billion-parameter model needs storage for half-precision weights, a full-precision master copy, optimizer variables, and gradients before activations are counted. He says activations, model parameters, and optimizer state account for large portions of the memory footprint, and sequence length is especially expensive because attention grows quadratically with the number of tokens. In the notebook, he caps examples at 256 tokens, pads shorter inputs, and cuts off longer ones to make the tensor shapes consistent.
Naive data parallelism can fail before training starts
Mark runs full fine-tuning with four 40-gigabyte A100 GPUs and shows a CUDA out-of-memory failure. The setup uses mixed precision and naive data parallelism, but the 20-billion-parameter Flan-UL2 model still cannot fit the model, optimizer, gradients, and activations into available memory. He explains that distributed training frameworks can use all four GPUs, yet distributing the work does not remove the total memory requirements. The example makes the infrastructure barrier concrete for teams that want to train large models locally or in a rented environment.
LoRA makes large-model fine-tuning practical by freezing most weights
The workshop switches to LoRA, or low-rank adaptation. The method freezes the pretrained weights and learns a smaller set of low-rank updates, mainly in attention layers. Mark describes this as parameter-efficient fine-tuning and says it cuts the memory footprint substantially in his example. The code change is small: configure the parameter-efficient fine-tuning settings, then wrap the model so gradients are disabled for the frozen parameters. Mark also mentions QLoRA, which applies four-bit quantization more broadly, but leaves it outside the workshop.
A managed platform targets the infrastructure last mile
Mark explains that Preemo plans to offer APIs for fine-tuning models through a managed platform. His reason is practical: producing a useful model requires more than selecting a base model. Teams must manage GPU capacity, batch size, sequence length, inference speed, and deployment. He says synthetic generation can already take time for one prompt, so repeating it tens of thousands of times creates another infrastructure problem. He also mentions Flash Attention as a possible optimization for reducing memory use. His view is that open-source models need better infrastructure because many developers cannot easily provision or tune the required machines.
"So what is our solution? I'm sure everybody has sort of heard about parameter efficient fine-tuning in the community these days, but the specific technique that we want to use is LoRA, which is a low rank adaptation method."46:49
Who should watch
You are deciding whether a hosted model is enough for a domain-specific product and need a concrete framework for choosing custom training.
You are building synthetic instruction data and want to combine generated examples with majority voting and diversity filtering.
You are blocked by GPU memory while fine-tuning a large model and want to understand what LoRA changes in the training setup.
