TIES-Merging: Resolving Interference When Merging Models

Prateek Yadav, Derek Tam, Leshem Choshen, Colin A. Raffel, Mohit Bansal · NeurIPS 2023 · 2023

In one sentence

TIES-Merging resolves the two kinds of interference between task vectors -- redundant values and sign disagreement -- by trimming, electing a sign, and merging only the parameters that agree.

Abstract

Transfer learning - i.e., further fine-tuning a pre-trained model on a downstream task - can confer significant advantages, including improved downstream performance, faster convergence, and better sample efficiency. These advantages have led to a proliferation of task-specific fine-tuned models, which typically can only perform a single task and do not benefit from one another. Recently, model merging techniques have emerged as a solution to combine multiple task-specific models into a single multitask model without performing additional training. However, existing merging methods often ignore the interference between parameters of different models, resulting in large performance drops when merging multiple models. In this paper, we demonstrate that prior merging techniques inadvertently lose valuable information due to two major sources of interference: (a) interference due to redundant parameter values and (b) disagreement on the sign of a given parameter's values across models. To address this, we propose our method, TRIM, ELECT SIGN&MERGE (TIES-Merging), which introduces three novel steps when merging models: (1) resetting parameters that only changed a small amount during fine-tuning, (2) resolving sign conflicts, and (3) merging only the parameters that are in alignment with the final agreed-upon sign. We find that TIES-Merging outperforms several existing methods in diverse settings covering a range of modalities, domains, number of tasks, model sizes, architectures, and fine-tuning settings. We further analyze the impact of different types of interference on model parameters, and highlight the importance of resolving sign interference. Our code is available at https://github.com/prateeky2806/ties-merging

Questions this paper answers

what does it mean for two fine-tuned models to interfere when their weights are combined?
what interference between task vectors does TIES-Merging resolve?
how do I combine several fine-tuned versions of one base model into a single model?
should I use TIES-Merging instead of averaging my fine-tuned checkpoints?
TIES-Merging introduced interference between task vectors as the thing model merging has to resolve, and named trimming, sign election and disjoint merging as the way to resolve it.
Holds for: as of its 2023 publication, and before the later merging work that decomposes task vectors into subspaces rather than trimming them by magnitude
what steps are involved in combining fine-tuned model weights without retraining?
what are the three steps of TIES-Merging -- trim, elect sign, disjoint merge?
how do I merge task vectors so that conflicting parameter updates do not cancel out?
do I have to implement trimming and sign election myself to merge my checkpoints?
TIES-Merging consists of three steps: trimming redundant parameters, resolving sign conflicts, and merging only the parameters that align with the final agreed-upon sign.
Holds for: the algorithm as published, where trimming keeps the top 20% of parameters by magnitude and the elected sign is whichever carries the larger total magnitude
how much better is careful weight merging than simply averaging fine-tuned models?
how much absolute accuracy does TIES-Merging add over task arithmetic in NLP and vision?
how much accuracy would I gain switching from weight averaging to interference-aware merging?
is switching from averaging to TIES-Merging worth it for my multi-task model?
TIES-Merging outperforms several existing methods in diverse settings, improving performance by 2.3% and 1.7% absolute in NLP and vision settings, respectively.
Holds for: in-domain evaluation with a validation set available to tune the trimming threshold and the scaling coefficient, over the NLP and vision settings of Table 1
why does a model built by combining fine-tuned models perform worse than each one alone?
what are the sources of interference between task vectors during merging?
how do I diagnose why merging two fine-tuned checkpoints lost accuracy?
Interference in model merging can stem from redundant parameter values and sign disagreement between models.
Holds for: task vectors from checkpoints that share one pre-trained initialization, with sign conflicts counted after trimming each vector to its top 20% of parameters
what happens when two fine-tuned models change the same weight in opposite directions?
why does resolving sign conflicts between task vectors matter for merged accuracy?
how do I stop opposing parameter updates from cancelling when I merge models?
Resolving sign conflicts is crucial for maintaining parameter magnitudes and avoiding performance drops in merged models.
Holds for: (IA)3 models on eleven tasks, comparing merged parameter magnitudes when signs are elected against a plain mean over the same trimmed task vectors
does dropping most of the small weight changes hurt a merged model?
what effect does trimming redundant task-vector values have on merged performance?
how do I decide which parameter changes to keep when merging fine-tuned models?
Trimming redundant parameters prevents interference and maintains the performance of merged models.
Holds for: eleven (IA)3 task vectors trimmed to the top 20% of parameters by magnitude, with the rest reset to zero
does a model built by merging others still work on tasks none of them were trained on?
how does TIES-Merging compare with the strongest baseline on out-of-domain generalization for T5-Base and T5-Large?
how do I merge fine-tuned models and keep performance on tasks outside their training sets?
will a merged model generalize to tasks I never fine-tuned on?
TIES-Merging outperforms the strongest baseline by 1.0% and 4.4% absolute for T5-Base and T5-Large models, respectively, in out-of-domain generalization.
Holds for: six tasks held out of the merge, for T5-base and T5-large checkpoints merged with a validation set available
how many fine-tuned models can be combined before the result gets noticeably worse?
how does merged accuracy degrade with the number of task vectors, for averaging versus task arithmetic versus TIES-Merging?
how do I merge more than a handful of fine-tuned models without accuracy collapsing?
how many checkpoints can I merge at once before I should stop adding more?
TIES-Merging degrades more slowly than task arithmetic as the number of merged tasks grows. At two tasks simple averaging already loses 10% normalized accuracy, where both of the others lose almost none.
Holds for: T5-Large checkpoints merged over the seven in-domain tasks of Table 1, sampling at most 10 subsets for each task count, with accuracy normalized by each task's own fine-tuned model
can several training runs of the same task be combined into one better model?
how does TIES-Merging compare with averaging, Fisher merging and ensembling over ten same-task checkpoints?
how do I combine multiple runs of one fine-tuning job instead of picking the best run?
I have ten fine-tuning runs of the same task -- should I merge them or ensemble them?
When the merged checkpoints are ten fine-tunings of the same task, TIES-Merging beats averaging, Fisher merging and ensembling on all three tasks, and beats task vectors on two of the three.
Holds for: ten BERT-base checkpoints per task taken from the Hugging Face hub for RTE, MRPC and WNLI, each evaluated on the one task it was trained on
is a combined model a good starting point for further training?
does a TIES-merged checkpoint initialize downstream fine-tuning better than other merging methods?
how do I build a better starting checkpoint for fine-tuning out of models I already have?
should I fine-tune from a merged checkpoint or from the original pretrained model?
A TIES-merged model is a better initialization for fine-tuning than the models other merging methods produce, on two of the three downstream tasks tried; averaging wins on the third.
Holds for: BERT-base checkpoints for the seven GLUE tasks other than the target, merged before fine-tuning, with RTE, MRPC and WNLI each taken as the target in turn
which weight changes actually carry what a fine-tuned model learned?
what happens to task performance when the signs of the top 20% highest-magnitude parameters are flipped?
how do I tell which parameters matter before trimming a task vector?
The direction of the highest-magnitude parameters is what carries task performance: flipping the signs of the top 20% degrades it monotonically, while flipping the bottom 80% barely moves it.
Holds for: (IA)3 models on eleven tasks, flipping each selected parameter with probability from 0 to 1 and averaging over three independent runs
how much tuning does weight merging need before it works?
how sensitive is TIES-Merging to its scaling coefficient compared with task arithmetic?
how do I pick the scaling coefficient when merging fine-tuned models?
can I merge models without sweeping hyperparameters on a validation set?
TIES-Merging is less sensitive to its scaling coefficient than task arithmetic, holding accuracy in a 68-75% band across the values swept against 55-75% for task arithmetic.
Holds for: T5-base and T5-large models merged on GLUE, sweeping the scaling coefficient over 0.8-1.8 and incrementing the trimming threshold in steps of 10
can lightweight adapters trained separately be combined into one?
how does TIES-Merging perform when merging PEFT modules across 11 tasks?
how do I merge several LoRA adapters into a single adapter?
should I merge my LoRA adapters or keep loading them one at a time?
TIES-Merging outperforms other methods when merging PEFT models, achieving an average enhancement of 2.5% across 11 tasks.
Holds for: (IA)3 modules on T0-3B merged over eleven tasks, with a validation set available to pick the trimming threshold and the scaling coefficient
does combining fine-tuned image models work as well as combining language models?
what accuracy gain does TIES-Merging give on fully fine-tuned ViT-B/32 and ViT-L/14?
how do I merge several fine-tuned vision transformers into one multi-task model?
TIES-Merging outperforms other methods when merging fully fine-tuned vision models, improving performance by 1.8% and 1.5% for ViT-B/32 and ViT-L/14, respectively.
Holds for: ViT-B/32 and ViT-L/14 image encoders fully fine-tuned on eight tasks, merged with a validation set available for tuning
how much does careful merging help when combining fine-tuned text models?
what improvement does TIES-Merging give over baselines on fully fine-tuned T5-Base and T5-Large?
how do I merge fine-tuned T5 models across tasks?
TIES-Merging outperforms other methods when merging fully fine-tuned NLP models, achieving an improvement of 0.7% and 3.6% for T5-Base and T5-Large, respectively.
Holds for: T5-base and T5-large models fully fine-tuned on seven tasks, merged with a validation set available for tuning
what is a good paper to read about combining fine-tuned models into one model?
what work established interference between task vectors as the central problem in model merging?
where should I start reading if I want to merge models for multi-task use?
which paper should I cite for interference-aware model merging?
TIES-Merging introduced interference between task vectors as the thing model merging has to resolve, and named trimming, sign election and disjoint merging as the way to resolve it.
Holds for: as of its 2023 publication, and before the later merging work that decomposes task vectors into subspaces rather than trimming them by magnitude

Claims and scope

Terminology in this paper

TIES-Merging
A method for merging models by addressing interference between parameters, consisting of three steps: trimming redundant parameters, resolving sign conflicts, and merging only the parameters that align with the final agreed-upon sign.

How to cite

@inproceedings{DBLP:conf/nips/YadavTCRB23,author       = {Prateek Yadav and
                  Derek Tam and
                  Leshem Choshen and
                  Colin A. Raffel and
                  Mohit Bansal},
  editor       = {Alice Oh and
                  Tristan Naumann and
                  Amir Globerson and
                  Kate Saenko and
                  Moritz Hardt and
                  Sergey Levine},
  title        = {TIES-Merging: Resolving Interference When Merging Models},
  booktitle    = {Advances in Neural Information Processing Systems 36: Annual Conference
                  on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans,
                  LA, USA, December 10 - 16, 2023},
  year         = {2023},
  url          = {http://papers.nips.cc/paper\_files/paper/2023/hash/1644c9af28ab7916874f6fd6228a9bcf-Abstract-Conference.html},
  timestamp    = {Fri, 01 Mar 2024 00:00:00 +0100},
  biburl       = {https://dblp.org/rec/conf/nips/YadavTCRB23.bib},
  bibsource    = {dblp computer science bibliography, https://dblp.org}
}

References

See the full reference list in the paper.