Knowledge is a Region in Weight Space for Fine-tuned Language Models

Almog Gueta; Elad Venezian; Colin Raffel; Noam Slonim; Yoav Katz; Leshem Choshen

doi:10.18653/V1/2023.FINDINGS-EMNLP.95

Knowledge is a Region in Weight Space for Fine-tuned Language Models

Almog Gueta, Elad Venezian, Colin Raffel, Noam Slonim, Yoav Katz, Leshem Choshen · Findings of EMNLP 2023 · 2023

In one sentence

Language models finetuned from the same pretrained checkpoint occupy compact, dataset- and task-specific regions of weight space whose interior points — including the region's centroid — perform as well as or better than the finetuned models that bound them.

Abstract

Research on neural networks has focused on understanding a single model trained on a single dataset. However, relatively little is known about the relationships between different models, particularly those trained or tested on different datasets. We address this by studying how the weight space and the underlying loss landscape of different models are interconnected. Specifically, we demonstrate that finetuned models that were optimized for high performance, reside in well-defined regions in weight space, and vice versa -- that any model that resides anywhere in those regions also exhibits high performance. Notably, we show that language models that have been finetuned on the same dataset form a tight cluster in the weight space, while models finetuned on different datasets from the same underlying task form a looser cluster. Moreover, traversing around the region between the models leads to new models that perform comparably or even better than models obtained via finetuning, even on tasks that the original models were not finetuned on. Our findings provide insight into the relationships between models, demonstrating that a model positioned between two similar models can acquire the knowledge of both. We leverage this and design a method for selecting a better model for efficient finetuning. Specifically, we show that starting from the center of the region is as effective, if not more, than using the pretrained model in 11 out of 12 datasets, resulting in an average accuracy improvement of 3.06.

Questions this paper answers

do two language models trained on the same dataset end up with similar weights?
can finetuned checkpoints be clustered by their training dataset using cosine distance between task vectors?
how do I tell which dataset or task a finetuned checkpoint was trained on from its weights alone?
if I only have the weights of a finetuned model, can I recover what data it was trained on?: RoBERTa-base models finetuned on the same dataset cluster together in weight space with 98% clustering accuracy. Distance was measured by cosine similarity between task vectors, with spectral clustering.
Holds for: 280 RoBERTa-base models, 20 seeds each over 12 GLUE/SuperGLUE datasets; all but 3 clusters matched perfectly. Euclidean distance gave no clear clusters.; Models finetuned on different datasets from the same task form a looser but still separable cluster in weight space. Grouping models from the NLI, Sentiment and Topic dataset families into 3 clusters reaches 90% clustering accuracy.
Holds for: RoBERTa-base, 5 seeds per dataset, English classification datasets in the NLI, Sentiment and Topic families; clustering by domain rather than task is much weaker.
why does averaging the weights of several finetuned models often work better than the models themselves?
why does weight averaging of finetuned models improve loss relative to the averaged endpoints?
how do I get a better model out of several finetuned checkpoints without any further training?
should I average my finetuned checkpoints, or just pick the best one?: Models sampled from the convex hull of 5 MNLI-finetuned models beat every model finetuned on other General datasets on MNLI loss 100% of the time. They also beat the MNLI-finetuned models themselves 88% of the time.
Holds for: RoBERTa-base encoders compared by generalized loss (a freshly trained linear probe per target dataset); interior models are uniformly sampled weighted averages of the 5 MNLI models.; Linearly interpolating between pairs of finetuned RoBERTa-base models yields models whose average loss is comparable to or lower than both endpoints, with the minimum often strictly between the two models.
Holds for: 5 MNLI models pairwise (10 pairs), MNLI with ESNLI (25 pairs, all NLI targets), MNLI with SST2 (25 pairs, all General targets); one shared pretrained checkpoint.; Knowledge is a Region in Weight Space offers a geometric explanation for why weight averaging and model-soup style fusion work. Averaging picks a point in the interior of a region, whereas finetuning tends to land on its boundary where loss is higher.
Holds for: Finetuning from a shared pretrained model on English classification data; an interpretive account of prior averaging results (model soups, Fisher merging, SWA) that the paper does not re-run.
are finetuned models sitting in the middle of a good-loss area of weight space or right at its edge?
how far can you extrapolate beyond finetuned endpoints before loss rises, and how wide is the low-loss basin?
how far can I push weights past two finetuned models before the model breaks?: Extrapolating past the finetuned endpoints raises loss rapidly at all 3 granularity levels, indicating that finetuned models sit near the edge of the low-loss region. The region has a relatively flat base and steep cliffs rather than being a broad subspace.
Holds for: 10 logarithmic steps of alpha from 1 to 32 and from 0 to -31, on the same MNLI, NLI and General model pairs used for interpolation; RoBERTa-base only. WNLI behaves differently from the other NLI datasets.
is there a better place to start finetuning from than the original pretrained model?
does initializing BitFit from the centroid of same-task finetuned models outperform pretrained initialization?
how do I improve parameter-efficient finetuning accuracy just by changing the starting weights?
I have several finetuned checkpoints already; should I start my next finetuning run from their average instead of the base model?: Starting BitFit finetuning from the centroid of finetuned models rather than from pretrained RoBERTa-base raises accuracy by 4.03 points on average across 12 datasets. The centroid wins on 9 datasets, ties on 2 and loses on 1 (WNLI, -1.41).
Holds for: For each target dataset the centroid excludes models finetuned on that dataset; BitFit parameter-efficient finetuning on 12 GLUE/SuperGLUE classification datasets from RoBERTa-base. Largest gain 11.19 on RTE.; In a few-shot setting capped at 1K training examples, BitFit from the region centroid gains 10.66 accuracy points on average over the pretrained model, reaching 33.99 on SST2 and 28.97 on MNLI.
Holds for: 12 GLUE/SuperGLUE classification datasets, RoBERTa-base, BitFit, centroid excluding models finetuned on the target dataset; WNLI still loses (-1.41) and MultiRC is flat (-0.06).
when there is very little training data, does starting from an average of existing finetuned models help?
how much does centroid initialization gain over pretrained initialization for BitFit in a few-shot regime?
how do I finetune a classifier with only about a thousand labelled examples and still get usable accuracy?
I only have a few hundred labelled examples; is a merged checkpoint a better starting point than the base model?: In a few-shot setting capped at 1K training examples, BitFit from the region centroid gains 10.66 accuracy points on average over the pretrained model, reaching 33.99 on SST2 and 28.97 on MNLI.
Holds for: 12 GLUE/SuperGLUE classification datasets, RoBERTa-base, BitFit, centroid excluding models finetuned on the target dataset; WNLI still loses (-1.41) and MultiRC is flat (-0.06).
does how much data you finetune on change where the model's weights end up, or only what the data is about?
is the direction of the task vector governed by training set content or by training set size?
if I finetune on 200 examples instead of 3000, will the resulting weights look like a different task?: The direction a finetuned model moves in weight space is determined by the type of training data, not its amount. Models trained on sub-samples of 200 to 3K examples cluster by dataset, and clustering by data size does not emerge.
Holds for: 9 General-family datasets with at least 3K training examples, sub-sampled at 200, 400, 800, 1.6K and 3K; RoBERTa-base. Clustering and data type agree in all but one case.
does the weight-space grouping of finetuned models depend on them all starting from the same base model?
is weight-space proximity of similarly finetuned models contingent on a shared pretrained initialization?
can I compare or merge finetuned models that came from two different pretraining runs of the same architecture?
my checkpoints come from different pretrained seeds; can I still expect them to land in the same region?: Weight-space proximity of similarly finetuned models is contingent on a shared pretrained initialization. Models finetuned on the same datasets from two different RoBERTa-base checkpoints cluster by which checkpoint they started from.
Holds for: Original RoBERTa-base and the independent re-implementation of Elazar et al. (2022), both finetuned on the General dataset family. Implies many equally good regions exist per ability, one reachable neighbourhood per starting point.
do finetuned models group together by the kind of text they were trained on, like tweets, the way they group by task?
does clustering finetuned models by domain, such as a Twitter group, work as well as clustering by task family?
how do I tell whether weight-space distance reflects the task a model does or the domain of text it saw?
can I use weight-space distance to find models trained on my domain, for example Twitter data?: Clustering finetuned models by domain rather than by task largely fails. Adding a Twitter-domain group to task-based clustering yields an F1 of only 30 on the Twitter cluster, against 100 on NLI, 61 on Topic and 71 on Sentiment.
Holds for: RoBERTa-base models, 4 clusters mapped 1-to-1 to the Twitter, NLI, Topic and Sentiment groups; task-only clustering of the same models reaches F1 100, 87 and 83.; Models finetuned on different datasets from the same task form a looser but still separable cluster in weight space. Grouping models from the NLI, Sentiment and Topic dataset families into 3 clusters reaches 90% clustering accuracy.
Holds for: RoBERTa-base, 5 seeds per dataset, English classification datasets in the NLI, Sentiment and Topic families; clustering by domain rather than task is much weaker.
which paper should I read first on the idea that finetuned models occupy whole regions of weight space rather than points?
what work extends linear mode connectivity to convex low-loss regions per dataset and per task?
where do I start reading to understand why weight averaging of finetuned language models works?: Knowledge is a Region in Weight Space reframes linear mode connectivity as region connectivity. It argues that finetuned language models bound a convex low-loss basin per dataset and per task, rather than merely lying on a low-loss line.
Holds for: English classification datasets with RoBERTa-base, as of 2023; extends prior connectivity work to models not trained on the same data, compared via a re-fit linear probe.; Knowledge is a Region in Weight Space offers a geometric explanation for why weight averaging and model-soup style fusion work. Averaging picks a point in the interior of a region, whereas finetuning tends to land on its boundary where loss is higher.
Holds for: Finetuning from a shared pretrained model on English classification data; an interpretive account of prior averaging results (model soups, Fisher merging, SWA) that the paper does not re-run.
is a model built by mixing several finetuned models any good on related datasets it never saw?
do interior points of a task's convex hull generalize better across that task's datasets than the finetuned models themselves?
how do I get a model that works across several NLI datasets without finetuning on each one?
should I use a point inside the region spanned by my finetuned models as a general-purpose model for that task?: Interior models of the NLI region beat models finetuned on non-NLI datasets in 100% of comparisons on NLI losses, versus 75.3% for the NLI-finetuned models. Interior models also beat those NLI-finetuned models 96.7% of the time.
Holds for: RoBERTa-base, NLI test datasets excluding ANLI, generalized loss with a per-target linear probe. In the General granularity the exterior group is norm-matched random perturbations of the pretrained model.; Models sampled from the convex hull of 5 MNLI-finetuned models beat every model finetuned on other General datasets on MNLI loss 100% of the time. They also beat the MNLI-finetuned models themselves 88% of the time.
Holds for: RoBERTa-base encoders compared by generalized loss (a freshly trained linear probe per target dataset); interior models are uniformly sampled weighted averages of the 5 MNLI models.

Claims and scope

RoBERTa-base models finetuned on the same dataset cluster together in weight space with 98% clustering accuracy. Distance was measured by cosine similarity between task vectors, with spectral clustering. (Section 4, Figure 2(a))
Scope: 280 RoBERTa-base models, 20 seeds each over 12 GLUE/SuperGLUE datasets; all but 3 clusters matched perfectly. Euclidean distance gave no clear clusters.
Models finetuned on different datasets from the same task form a looser but still separable cluster in weight space. Grouping models from the NLI, Sentiment and Topic dataset families into 3 clusters reaches 90% clustering accuracy. (Section 4, Figure 2(b))
Scope: RoBERTa-base, 5 seeds per dataset, English classification datasets in the NLI, Sentiment and Topic families; clustering by domain rather than task is much weaker.
Clustering finetuned models by domain rather than by task largely fails. Adding a Twitter-domain group to task-based clustering yields an F1 of only 30 on the Twitter cluster, against 100 on NLI, 61 on Topic and 71 on Sentiment. (Table 1, Appendix D)
Scope: RoBERTa-base models, 4 clusters mapped 1-to-1 to the Twitter, NLI, Topic and Sentiment groups; task-only clustering of the same models reaches F1 100, 87 and 83.
The direction a finetuned model moves in weight space is determined by the type of training data, not its amount. Models trained on sub-samples of 200 to 3K examples cluster by dataset, and clustering by data size does not emerge. (Section 4.1, Figures 7 and 8)
Scope: 9 General-family datasets with at least 3K training examples, sub-sampled at 200, 400, 800, 1.6K and 3K; RoBERTa-base. Clustering and data type agree in all but one case.
Models sampled from the convex hull of 5 MNLI-finetuned models beat every model finetuned on other General datasets on MNLI loss 100% of the time. They also beat the MNLI-finetuned models themselves 88% of the time. (Section 5.2, Figure 4(a))
Scope: RoBERTa-base encoders compared by generalized loss (a freshly trained linear probe per target dataset); interior models are uniformly sampled weighted averages of the 5 MNLI models.
Interior models of the NLI region beat models finetuned on non-NLI datasets in 100% of comparisons on NLI losses, versus 75.3% for the NLI-finetuned models. Interior models also beat those NLI-finetuned models 96.7% of the time. (Section 5.2, Figure 4)
Scope: RoBERTa-base, NLI test datasets excluding ANLI, generalized loss with a per-target linear probe. In the General granularity the exterior group is norm-matched random perturbations of the pretrained model.
Linearly interpolating between pairs of finetuned RoBERTa-base models yields models whose average loss is comparable to or lower than both endpoints, with the minimum often strictly between the two models. (Section 5.1, Figures 3 and 10)
Scope: 5 MNLI models pairwise (10 pairs), MNLI with ESNLI (25 pairs, all NLI targets), MNLI with SST2 (25 pairs, all General targets); one shared pretrained checkpoint.
Extrapolating past the finetuned endpoints raises loss rapidly at all 3 granularity levels, indicating that finetuned models sit near the edge of the low-loss region. The region has a relatively flat base and steep cliffs rather than being a broad subspace. (Section 6.1, Figures 5 and 12)
Scope: 10 logarithmic steps of alpha from 1 to 32 and from 0 to -31, on the same MNLI, NLI and General model pairs used for interpolation; RoBERTa-base only. WNLI behaves differently from the other NLI datasets.
Starting BitFit finetuning from the centroid of finetuned models rather than from pretrained RoBERTa-base raises accuracy by 4.03 points on average across 12 datasets. The centroid wins on 9 datasets, ties on 2 and loses on 1 (WNLI, -1.41). (Table 2, Figure 6)
Scope: For each target dataset the centroid excludes models finetuned on that dataset; BitFit parameter-efficient finetuning on 12 GLUE/SuperGLUE classification datasets from RoBERTa-base. Largest gain 11.19 on RTE.
In a few-shot setting capped at 1K training examples, BitFit from the region centroid gains 10.66 accuracy points on average over the pretrained model, reaching 33.99 on SST2 and 28.97 on MNLI. (Table 3, Figure 14)
Scope: 12 GLUE/SuperGLUE classification datasets, RoBERTa-base, BitFit, centroid excluding models finetuned on the target dataset; WNLI still loses (-1.41) and MultiRC is flat (-0.06).
Weight-space proximity of similarly finetuned models is contingent on a shared pretrained initialization. Models finetuned on the same datasets from two different RoBERTa-base checkpoints cluster by which checkpoint they started from. (Appendix B)
Scope: Original RoBERTa-base and the independent re-implementation of Elazar et al. (2022), both finetuned on the General dataset family. Implies many equally good regions exist per ability, one reachable neighbourhood per starting point.
Knowledge is a Region in Weight Space reframes linear mode connectivity as region connectivity. It argues that finetuned language models bound a convex low-loss basin per dataset and per task, rather than merely lying on a low-loss line. (Section 1, Section 8)
Scope: English classification datasets with RoBERTa-base, as of 2023; extends prior connectivity work to models not trained on the same data, compared via a re-fit linear probe.
Knowledge is a Region in Weight Space offers a geometric explanation for why weight averaging and model-soup style fusion work. Averaging picks a point in the interior of a region, whereas finetuning tends to land on its boundary where loss is higher. (Section 8, Section 9)
Scope: Finetuning from a shared pretrained model on English classification data; an interpretive account of prior averaging results (model soups, Fisher merging, SWA) that the paper does not re-run.

Common misreadings

The claim is not that any two finetuned models can be averaged: proximity and the low-loss region depend on a shared pretrained initialization, and models finetuned from two different RoBERTa-base checkpoints cluster by checkpoint instead.
The low-loss region is not an unbounded subspace. Extrapolating beyond the finetuned endpoints degrades performance quickly, so the region is a small basin with steep edges.
The centroid initialization result is not evidence that averaging always helps: on WNLI the centroid underperformed the pretrained model by 1.41 points, and on MultiRC it was flat.
The results cover English classification datasets with RoBERTa-base finetuning; they were not tested on randomly initialized networks, generative tasks, or many pretrained backbones.
The 4.03-point average gain is not obtained by evaluating the averaged model directly; it is the gain after subsequent BitFit finetuning on the target dataset.

Terminology in this paper

generalized loss: The loss of a finetuned encoder on a target dataset after discarding its original head and training a fresh linear probe on that target's training data, which makes models finetuned on different datasets comparable.
In, In', Ex: Three model groups compared in weight-space experiments: In are models finetuned on datasets sharing a trait, In' are weighted averages sampled from the convex hull of In, and Ex are models not sharing that trait (or random perturbations of the pretrained model at matched norm).
PB: The probability that a randomly chosen model from one group attains a lower generalized loss than a randomly chosen model from another group.
centroid: The uniform average of the weights of a set of models finetuned from the same pretrained checkpoint, taken as a representative interior point of their weight-space region.
task vector: The difference between a finetuned model's weights and the pretrained weights it started from, used as the representation whose cosine similarity defines model distance.

How to cite

@inproceedings{DBLP:conf/emnlp/GuetaVRSKC23,author       = {Almog Gueta and
                  Elad Venezian and
                  Colin Raffel and
                  Noam Slonim and
                  Yoav Katz and
                  Leshem Choshen},
  editor       = {Houda Bouamor and
                  Juan Pino and
                  Kalika Bali},
  title        = {Knowledge is a Region in Weight Space for Fine-tuned Language Models},
  booktitle    = {Findings of the Association for Computational Linguistics: {EMNLP}
                  2023, Singapore, December 6-10, 2023},
  pages        = {1350--1370},
  publisher    = {Association for Computational Linguistics},
  year         = {2023},
  url          = {https://doi.org/10.18653/v1/2023.findings-emnlp.95},
  doi          = {10.18653/V1/2023.FINDINGS-EMNLP.95},
  timestamp    = {Fri, 12 Apr 2024 01:00:00 +0200},
  biburl       = {https://dblp.org/rec/conf/emnlp/GuetaVRSKC23.bib},
  bibsource    = {dblp computer science bibliography, https://dblp.org}
}

References

See the full reference list in the paper.