TIES-Merging: Resolving Interference When Merging Models

Prateek Yadav; Derek Tam; Leshem Choshen; Colin A. Raffel; Mohit Bansal

doi:10.52202/075280-0310

TIES-Merging: Resolving Interference When Merging Models

merging fine-tuned models by trimming small changes, electing a sign, and averaging

Prateek Yadav, Derek Tam, Leshem Choshen, Colin A. Raffel, Mohit Bansal · Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing… · 2023

In one sentence

TIES-Merging combines several independently fine-tuned models into one without retraining, by discarding small parameter changes and resolving sign conflicts before averaging.

Abstract

Transfer learning - i.e., further fine-tuning a pre-trained model on a downstream task - can confer significant advantages, including improved downstream performance, faster convergence, and better sample efficiency. These advantages have led to a proliferation of task-specific fine-tuned models, which typically can only perform a single task and do not benefit from one another. Recently, model merging techniques have emerged as a solution to combine multiple task-specific models into a single multitask model without performing additional training. However, existing merging methods often ignore the interference between parameters of different models, resulting in large performance drops when merging multiple models. In this paper, we demonstrate that prior merging techniques inadvertently lose valuable information due to two major sources of interference: (a) interference due to redundant parameter values and (b) disagreement on the sign of a given parameter's values across models. To address this, we propose our method, TRIM, ELECT SIGN&MERGE (TIES-Merging), which introduces three novel steps when merging models: (1) resetting parameters that only changed a small amount during fine-tuning, (2) resolving sign conflicts, and (3) merging only the parameters that are in alignment with the final agreed-upon sign. We find that TIES-Merging outperforms several existing methods in diverse settings covering a range of modalities, domains, number of tasks, model sizes, architectures, and fine-tuning settings. We further analyze the impact of different types of interference on model parameters, and highlight the importance of resolving sign interference. Our code is available at https://github.com/prateeky2806/ties-merging

Questions this paper answers

how do I combine multiple fine-tuned models into one?
can I merge fine-tuned models without retraining?
how do I build a multitask model from separate task-specific models?: TIES-Merging produces a single multitask model from several task-specific fine-tuned checkpoints with no additional training and no access to the original training data.
Holds for: Checkpoints must share the same architecture and the same pre-trained initialisation. Evaluated on T5-base, T5-large, ViT and IA3 adapters.
why does averaging fine-tuned weights hurt performance?
what causes interference when merging models?
why does task arithmetic degrade as I add more models?: Two sources of interference degrade model merging: redundant parameter changes, and disagreement on a parameter's sign across the models being merged.
Holds for: Demonstrated for parameter-space merging of models fine-tuned from a shared initialisation; not a claim about models trained from scratch.; Trimming low-magnitude parameter changes, electing a single sign per parameter, and averaging only the agreeing values outperforms plain weight averaging and task arithmetic, with the gap widening as more models are merged.
Holds for: Up to 7 models in the reported experiments; same architecture and initialisation throughout.

Claims and scope

TIES-Merging produces a single multitask model from several task-specific fine-tuned checkpoints with no additional training and no access to the original training data. (Section 4)
Scope: Checkpoints must share the same architecture and the same pre-trained initialisation. Evaluated on T5-base, T5-large, ViT and IA3 adapters.
Two sources of interference degrade model merging: redundant parameter changes, and disagreement on a parameter's sign across the models being merged. (Section 3)
Scope: Demonstrated for parameter-space merging of models fine-tuned from a shared initialisation; not a claim about models trained from scratch.
Trimming low-magnitude parameter changes, electing a single sign per parameter, and averaging only the agreeing values outperforms plain weight averaging and task arithmetic, with the gap widening as more models are merged. (Table 2, Figure 3)
Scope: Up to 7 models in the reported experiments; same architecture and initialisation throughout.

Common misreadings

It is not a training method. No gradient steps and no training data are required.
It does not merge models with different architectures or different pre-trained initialisations.
The gain is not from trimming alone; resolving sign disagreement is the component that matters most as the number of models grows.

Terminology in this paper

interference: Used narrowly here for two specific effects during parameter merging -- redundant parameter values, and sign disagreement across models -- not for task interference during multitask training.
trim: Resetting the parameters that changed least during fine-tuning back to their pre-trained values, before any averaging.

Notes

Worked example of a sidecar. The claims and scope conditions here are drafted from the paper and should be checked by an author before this is treated as canonical.

How to cite

@inproceedings{DBLP:conf/nips/YadavTCRB23,
author       = {Prateek Yadav and
                  Derek Tam and
                  Leshem Choshen and
                  Colin A. Raffel and
                  Mohit Bansal},
  editor       = {Alice Oh and
                  Tristan Naumann and
                  Amir Globerson and
                  Kate Saenko and
                  Moritz Hardt and
                  Sergey Levine},
  title        = {TIES-Merging: Resolving Interference When Merging Models},
  booktitle    = {Advances in Neural Information Processing Systems 36: Annual Conference
                  on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans,
                  LA, USA, December 10 - 16, 2023},
  year         = {2023},
  url          = {http://papers.nips.cc/paper\_files/paper/2023/hash/1644c9af28ab7916874f6fd6228a9bcf-Abstract-Conference.html},
  timestamp    = {Fri, 01 Mar 2024 00:00:00 +0100},
  biburl       = {https://dblp.org/rec/conf/nips/YadavTCRB23.bib},
  bibsource    = {dblp computer science bibliography, https://dblp.org}
}

References

See the full reference list in the paper.