Fusing finetuned models for better pretraining

averaging the weights of several finetuned models to create a new base model for finetuning on a fresh task

Leshem Choshen, Elad Venezian, Noam Slonim, Yoav Katz · arXiv · 2022

In one sentence

Fusing averages the weights of several existing finetuned models into a new base model that finetunes better and more stably than the pretrained model it came from, at almost no cost and without access to any source data.

Abstract

Pretrained models are the standard starting point for training. This approach consistently outperforms the use of a random initialization. However, pretraining is a costly endeavour that few can undertake. In this paper, we create better base models at hardly any cost, by fusing multiple existing fine tuned models into one. Specifically, we fuse by averaging the weights of these models. We show that the fused model results surpass the pretrained model ones. We also show that fusing is often better than intertraining. We find that fusing is less dependent on the target task. Furthermore, weight decay nullifies intertraining effects but not those of fusing.

Questions this paper answers

if I average together the weights of several models that were each fine-tuned on a different task, is the result a better starting point for training on a new task than the original pretrained model?
does weight averaging of multiple finetuned checkpoints yield a stronger initialization for downstream finetuning than the pretrained backbone?
how do I build a better base checkpoint for a new task out of finetuned models I already have lying around?
should I start my next finetuning run from an average of my old finetuned checkpoints instead of the pretrained release?
Averaging the weights of several finetuned T5v1.1-small models yields a better base model for finetuning than the pretrained model itself. Average accuracy across 3 target dataset families rises from 62.40 to 64.72 when the NLI models are fused.
Holds for: T5v1.1-small, AdamW without weight decay, 30 English text-classification datasets in 3 families (GLUE/SuperGLUE, NLI, TweetEval); the target task is always excluded from the source tasks; averaged over 5 seeds.
Fusing gives more stable finetuning than starting from the pretrained model: the standard deviation of target accuracy averages 1.21 to 2.27 across fused source families against 3.64 for the pretrained base model.
Holds for: T5v1.1-small, 5 random seeds, 3 target dataset families; intertraining is comparably stable (1.61 to 2.24), so the stability gain is over the pretrained baseline and not over intertraining.
is averaging a bunch of models trained on other tasks as good as first training on one carefully picked helper task?
does fusing all available finetuned checkpoints outperform intermediate-task training on a well-selected source task?
how do I choose between averaging many source checkpoints and running a single intermediate-task training stage before my target task?
I already know which helper task transfers well to my target task, so is averaging models worth doing instead?
Fusing all available finetuned models does not beat carefully chosen intertraining on T5v1.1-small. Intertraining on the general dataset family averages 66.48 accuracy over 3 target families, against 64.72 for the best fused set and 62.40 for the pretrained baseline.
Holds for: Fusing all available models indiscriminately, with the intertraining baseline picking the model finetuned on the largest source dataset (MNLI for the general family, a known strong intertraining task); 5 seeds; no weight decay.
Fusing 2 finetuned models is often better than intertraining with the better of the two. In all but 1 of the GLUE model pairs, fusing beats intertraining with the worse of the two.
Holds for: Pairs of models finetuned on GLUE datasets, evaluated as base models on GLUE targets, T5v1.1-small, no weight decay; improvement measured against the pretrained baseline.
when averaging fine-tuned models to make a starting point, does it matter which models go into the average?
which source checkpoints should be selected for weight fusion, and are the best fusion candidates the same as the best intermediate-task donors?
how do I pick which finetuned GLUE checkpoints to average into a base model for a new task?
can I just throw every finetuned checkpoint I have into the average, or do I need to select the pair?
Fusing the MNLI and SST2 finetuned models reaches the highest accuracy in the GLUE fusing experiment, and the models most useful for intertraining are also the best models to fuse.
Holds for: GLUE target datasets, T5v1.1-small, no weight decay; MNLI and SST2 are the 2 largest general-family training sets and MNLI is a previously reported strong intertraining source.
Intertraining is sensitive to which target task it is applied to while fusing is not. The best intertraining source family is the one most similar to the target, whereas the NLI model set is the best set to fuse for all 3 target families.
Holds for: 3 dataset families as both sources and targets (general GLUE/SuperGLUE, NLI, Twitter), T5v1.1-small, no weight decay, 5 seeds.
does averaging finetuned models help every new task equally, or do I have to redo the choice of models for each new task?
is fused-checkpoint initialization less target-task sensitive than intermediate-task transfer?
how do I avoid re-selecting a source task every time I move to a new target task?
can one averaged base model serve all my target tasks, or must I tune source selection per task?
Intertraining is sensitive to which target task it is applied to while fusing is not. The best intertraining source family is the one most similar to the target, whereas the NLI model set is the best set to fuse for all 3 target families.
Holds for: 3 dataset families as both sources and targets (general GLUE/SuperGLUE, NLI, Twitter), T5v1.1-small, no weight decay, 5 seeds.
does using weight decay while finetuning wipe out the gains from training on a helper task first?
how does AdamW weight decay during finetuning interact with the benefit of intermediate-task training versus checkpoint fusion?
how do I keep the advantage of a better starting checkpoint when my finetuning recipe uses weight decay?
my finetuning recipe uses weight decay of 0.01, so will a fused starting point still help me?
Weight decay of 0.01 in finetuning nullifies the benefit of intertraining but not of fusing. On general target datasets intertraining drops from 72.76 to 61.7, level with the pretrained baseline's 61.6, while fusing falls only from 68.12 to 65.1.
Holds for: T5v1.1-small finetuned with AdamW, decay 0.01 versus none, general (GLUE/SuperGLUE) target datasets; absolute accuracies are lower with decay overall. Initial BERT trials, pretrained with decay, showed less of the adverse effect.
does starting from an averaged model make training results vary less from one random seed to another?
does initializing from fused finetuned weights reduce across-seed standard deviation of target-task accuracy?
how do I cut the seed-to-seed variance in my finetuning runs?
my finetuning results swing a lot between seeds, would starting from averaged checkpoints steady them?
Fusing gives more stable finetuning than starting from the pretrained model: the standard deviation of target accuracy averages 1.21 to 2.27 across fused source families against 3.64 for the pretrained base model.
Holds for: T5v1.1-small, 5 random seeds, 3 target dataset families; intertraining is comparably stable (1.61 to 2.24), so the stability gain is over the pretrained baseline and not over intertraining.
how much training data do the donor models need before averaging them makes a useful starting point?
how does source-task training set size affect the quality of a fused base model relative to intermediate-task training?
how do I tell whether my source checkpoints were trained on enough data to be worth averaging?
my source tasks only have small datasets, is averaging their checkpoints still worth it?
More source training data produces better fused base models, and fusing gains from smaller amounts of source data than intertraining while its improvement also plateaus earlier.
Holds for: General target datasets, T5v1.1-small, source data amounts swept on a log scale; the earlier-plateau trend is described in the paper as noisier than the main monotone trend.
what research first suggested making a better starting model by averaging models people had already fine-tuned?
which work proposed recycling finetuned checkpoints via weight averaging as a replacement for the pretrained initialization in transfer learning?
where should I start reading about reusing existing finetuned checkpoints instead of pretraining a new base model?
what should I read first if I want to build base models out of finetuned checkpoints I already have?
Fusing finetuned models for better pretraining proposes reversing the transfer-learning pipeline. Rather than reusing a pretrained model to make finetuned models, it recycles existing finetuned models by weight averaging into a better base model for new target tasks.
Holds for: As of the 2022 arXiv preprint; demonstrated only on T5v1.1-small and English text classification. Contemporaneous weight-averaging work (Model Soups, Fisher-weighted averaging) fuses models for direct use on a task rather than as an initialization for new tasks.
Fusing finetuned models for better pretraining frames intertraining as the special case of fusing a single model. Choosing a base model becomes a question of which set of finetuned models to average rather than which one to pick.
Holds for: 3 dataset families and 30 English classification datasets; all fused models share one pretrained initialization, and no theory for why averaging helps is offered.
what do you actually need to have on hand to turn a pile of fine-tuned models into a better starting model?
what assumptions does fusing finetuned checkpoints into a base model make about access to source training data and multitask pretraining compute?
how do I improve a base model when the source datasets are unavailable and I cannot afford multitask pretraining?
I only have the finetuned weights and no source data or pretraining budget, can I still get a better base model?
Fusing finetuned models for better pretraining assumes access only to finetuned model weights. Neither the source training data nor the compute for massively multitask pretraining is required, which is what makes averaging rather than retraining the mechanism.
Holds for: As of the 2022 preprint; the assumption rules out data-dependent alternatives such as Fisher-information weighting. Relevance depends on finetuned models being shared, estimated only indirectly from HuggingFace hub counts and 20 sampled EMNLP 2021 papers.
Fusing finetuned models for better pretraining proposes reversing the transfer-learning pipeline. Rather than reusing a pretrained model to make finetuned models, it recycles existing finetuned models by weight averaging into a better base model for new target tasks.
Holds for: As of the 2022 arXiv preprint; demonstrated only on T5v1.1-small and English text classification. Contemporaneous weight-averaging work (Model Soups, Fisher-weighted averaging) fuses models for direct use on a task rather than as an initialization for new tasks.
how does picking one fine-tuned model to start from relate to averaging several of them?
is intermediate-task training the single-model special case of weight fusion for initialization?
how should I frame the choice of a starting checkpoint if averaging several sources is on the table?
should I think of choosing a helper task and averaging several helper models as the same decision?
Fusing finetuned models for better pretraining frames intertraining as the special case of fusing a single model. Choosing a base model becomes a question of which set of finetuned models to average rather than which one to pick.
Holds for: 3 dataset families and 30 English classification datasets; all fused models share one pretrained initialization, and no theory for why averaging helps is offered.

Claims and scope

Common misreadings

Terminology in this paper

Fusing
Combining several models finetuned from the same pretrained initialization into a single new base model by averaging each shared weight, and then finetuning that base model on a target task.
Intertraining
Using a model already finetuned on some other source task as the initialization for finetuning on a target task, rather than starting from the pretrained model.
Available finetuned models
The set of models assumed to already exist for reuse in an experiment, simulated by finetuning the pretrained model on each source task, always excluding the target task.

How to cite

@article{Choshen2022FusingFM,title={Fusing finetuned models for better pretraining},
  author={Leshem Choshen and Elad Venezian and Noam Slonim and Yoav Katz},
  journal={ArXiv},
  year={2022},
  volume={abs/2204.03044},
  url={https://api.semanticscholar.org/CorpusID:248006458}
}

References

See the full reference list in the paper.