Model merging can combine fine-tuned models into stronger open models at low compute cost, and many leading 7B models on the Open LLM Leaderboard were merges.
2
SLERP, TIES, passthrough, and Franken-merging combine model weights in different ways, with TIES reducing redundant parameters and passthrough adding layers.
3
Mixture-of-experts models activate only part of a larger network, while mergekit and the Lazy Merge Kit notebook make model creation and sharing accessible.
Summary
Maxime Labonne explains why model merging has become common in the open-source LLM community. He covers SLERP, TIES, passthrough merging, and Franken-merging, then describes a mixture-of-experts approach that combines feed-forward layers from several models behind a router. Merging usually needs only a CPU, but evaluating the resulting models is expensive because they must be run across several benchmarks. Labonne recommends using multiple evaluations rather than relying on one leaderboard. He also describes direct preference optimization as a relatively inexpensive way to improve a merged model or add task-specific behavior, although it requires fresh preference data. The practical path he recommends is mergekit, which uses shareable YAML configuration files, or the Lazy Merge Kit Colab notebook for a simpler workflow. The talk is a compact guide to experimenting with merged models, while remaining honest that benchmark contamination and limited evaluation make model quality difficult to judge.
Labonne points to the Open LLM Leaderboard as evidence that merging is producing competitive models. When he checked the 7B models, he says all of them were merges. He accepts that test-set overfitting and contamination affect the result, but says the models also perform well beyond that issue. This gives merging a practical reason to exist: open-source developers can combine capabilities from existing fine-tuned models instead of training a new model from scratch.
SLERP interpolates two models with layer-specific control
SLERP means spherical linear interpolation. It averages model weights while preserving properties of the space in which those weights sit. The method can merge only two models at a time, but users can set interpolation factors for different layer types and gradients. Labonne says this permits precise experiments, although changing the base parameters slightly often has little effect on the final model's performance. He gives Big Old 147B as an example and says its configuration is available for reproduction.
TIES reduces redundant task information before combining models
TIES combines two techniques intended to reduce redundancy in model parameters. Pruning resets selected fine-tuned weights to their original base-model values, while the process keeps only the top percentage of significant parameters. The method then rescales weights across models and handles their signs before combining them. Labonne describes the result as a task vector that captures a model's learned knowledge in a form that can be combined efficiently. Unlike SLERP, TIES can merge multiple models.
Passthrough and Franken-merging add depth to existing models
Passthrough merging concatenates layers from different language models or from the same model. Labonne calls the approach experimental, but says larger models made this way can gain reasoning abilities and answer questions their source models could not answer. These Franken-merges can become very large, including examples around 120 billion and 155 billion parameters. Running them can require high-end hardware such as an NVIDIA 3090 or 4090.
Franken mixtures of experts trade storage for selective computation
A mixture-of-experts model routes each input through only part of its network, so it does not activate every expert for every token. This can improve efficiency and performance because the overall model contains more parameters. The trade-off is that fine-tuning is difficult and the complete network still has to fit in memory, even when only a subnetwork is active. Labonne describes combining the feed-forward layers of several models and adding a router as one practical construction.
mergekit turns merging recipes into shareable configurations
Labonne presents mergekit, created by Charles G., as the library supporting the techniques in the talk. It uses YAML configuration files, which let users describe a merge, share the recipe, and reproduce it. This matters because the merge itself can run on a CPU and does not require a GPU. The expensive part begins afterward, when the resulting model must be loaded and evaluated.
Merged models need several evaluations because no single leaderboard is enough
Labonne says evaluation is the hardest and most costly part of model merging. He recommends using several benchmarks to get a better picture of quality, since the Open LLM Leaderboard has limitations and models can overfit evaluations. He mentions AGI evolve, EQ Bench, and MT Bench as additional ways to assess models, including conversational performance. His point is practical: a merge recipe is easy to run, but deciding whether it produced a useful model requires broader testing.
DPO can refine a merge or add a task-specific behavior
After merging, users can apply direct preference optimization. Labonne describes DPO as an inexpensive way to improve a merged model without starting over. Preference data can also add a behavior or steer the model toward a type of conversation. The limitation is the need for new preference datasets. He says this process is intended to make models better, rather than censor them.
The Lazy Merge Kit notebook lowers the barrier to experimentation
Labonne recommends his article and the Lazy Merge Kit Colab notebook for people who want to try merging. The notebook wraps the process so users specify a configuration and click one button. This provides a simpler starting point than assembling the full workflow themselves, while still producing models that can be uploaded and shared through the Hugging Face Hub.
