Model merging with SVD to tie the Knots

George Stoica, Pratik Ramesh, B. Ecsedi, Leshem Choshen, J. Hoffman · ICLR 2025 · 2025

In one sentence

KnOTS takes a joint SVD over the LoRA updates of separately finetuned models so that existing mergers work in one shared basis, and tests the result on a joint-task benchmark that asks whether the merge is general.

Abstract

Recent model merging methods demonstrate that the parameters of fully-finetuned models specializing in distinct tasks can be combined into one model capable of solving all tasks without retraining. Yet, this success does not transfer well when merging LoRA finetuned models. We study this phenomenon and observe that the weights of LoRA finetuned models showcase a lower degree of alignment compared to their fully-finetuned counterparts. We hypothesize that improving this alignment is key to obtaining better LoRA model merges, and propose KnOTS to address this problem. KnOTS uses the SVD to jointly transform the weights of different LoRA models into an aligned space, where existing merging methods can be applied. In addition, we introduce a new benchmark that explicitly evaluates whether merged models are general models. Notably, KnOTS consistently improves LoRA merging by up to 4.3% across several vision and language benchmarks, including our new setting. We release our code at: https://github.com/gstoica27/KnOTS.

Questions this paper answers

why do separately fine-tuned adapters get worse when you average them together, and what fixes it?
how does a joint SVD of LoRA task updates improve multi-task merging accuracy over merging in each adapter's own basis?
how do I combine several LoRA adapters into one model so the combined model keeps their accuracy?
I have several LoRA fine-tunes of the same base model — will aligning their updates before merging actually buy me accuracy?
KnOTS uses SVD to align LoRA model updates, improving merging performance by up to 4.3% across vision and language benchmarks.
Holds for: LoRA-finetuned models merged with TIES or DARE-TIES, where the 4.3% is KnOTS-TIES against TIES on the eight-task ViT-B/32 vision benchmark
how do you test whether a single merged model is any good when you do not know which task an input came from?
what evaluation setting measures merged-model generality over the union of label spaces from all constituent datasets?
how do I evaluate a merged multi-task model without telling it which dataset each test input belongs to?
should I be benchmarking my merged checkpoint on each dataset separately, or on all of the labels at once?
KnOTS introduces a new benchmark that evaluates the generality of merged models by merging models on the union of all inputs and labels across multiple datasets.
Holds for: the eight vision datasets pooled into one label space, so the merged model has to pick a label across all of them rather than within one
what did the KnOTS work establish about combining low-rank fine-tunes of one base model?
what is the KnOTS contribution to LoRA merging, and what did it show about the bases of different adapters' updates?
if I only read one thing before merging LoRA adapters, what does KnOTS tell me that I need to know?
KnOTS is the reference point for merging LoRA adapters. It showed that different adapters' updates sit in different bases, and that one joint SVD is enough to bring them into a shared one.
Holds for: as of its 2025 publication, and for LoRA adapters specifically rather than the fully finetuned models earlier merging work addressed
does the trick for merging low-rank fine-tunes still help when the vision backbone gets bigger?
how does KnOTS-TIES scale from ViT-B/32 to ViT-L/14 LoRA finetunes on 8-task merging?
how do I merge eight LoRA-finetuned vision transformers when the backbone is a large CLIP model?
I am merging adapters on a large ViT rather than a small one — will the gain hold at that scale?
KnOTS scales to larger vision models, improving TIES by 3% when the models being merged are eight ViT-L/14 LoRA finetunes rather than ViT-B/32.
Holds for: eight ViT-L/14 CLIP models LoRA-finetuned on the same vision datasets as the ViT-B/32 setting, scored as normalized per-task accuracy
does aligning adapter updates before merging work on large language models, not just image models?
how does KnOTS-TIES compare with task arithmetic, TIES and DARE-TIES when merging Llama3-8B LoRA finetunes on NLI datasets?
how do I merge several 8B-parameter LoRA fine-tunes trained on different natural language inference datasets?
I have six LoRA fine-tunes of Llama3-8B on different NLI data — which merging method should I pick?
KnOTS-TIES outperforms task arithmetic, TIES and DARE-TIES by up to 2.9% normalized accuracy when merging six Llama3-8B models LoRA-finetuned on different NLI datasets.
Holds for: normalized against the 92.9% average per-task accuracy of the six individual finetuned models, with the merge itself tuned on no held-out data
does merging more fine-tuned models at once wipe out the advantage of aligning their updates first?
how does the KnOTS-TIES margin over TIES and task arithmetic behave as the number of merged tasks increases?
how many LoRA adapters can I merge into one model before the alignment step stops paying off?
I want one model covering many tasks, not two or three — does the gap hold as I add more adapters?
KnOTS-TIES holds a gap of more than 4% over TIES and task arithmetic once more than two tasks are merged, and the gap does not close as the count grows.
Holds for: ViT-B/32 models on the eight-task vision benchmark, sweeping the number of merged tasks and scoring average normalized accuracy
does the rank you fine-tuned at change whether aligning updates before merging helps?
is the KnOTS-TIES advantage over TIES stable across LoRA ranks from 4 up to the full feature dimension?
how do I choose a LoRA rank for adapters I plan to merge later?
my adapters were trained at a low rank — do I need to retrain at higher rank to get the merging benefit?
KnOTS-TIES outperforms TIES at every LoRA rank tried, from rank 4 up to the full-rank 768 setting where rank equals the model's feature dimension.
Holds for: ViT-B/32 models LoRA-finetuned at ranks 4, 16, 64, 256 and 768, merged on the eight-task per-task vision benchmark
when you stack the weight changes from several fine-tunes before factorizing them, does the direction you stack them in matter?
why does column-wise concatenation of LoRA task updates before the SVD outperform the row-wise variant in KnOTS?
which way should I concatenate task updates before running an SVD to align them for merging?
Concatenating the task updates column-wise before the SVD is what makes KnOTS work: the row-wise variant performs 2.6% worse.
Holds for: ViT-B/32 models on the eight-task vision benchmark, with the concatenation order the only thing varied between the two runs
can one merged model handle inputs from all of its source tasks at once without being told which task an input came from?
how does KnOTS-TIES compare with task arithmetic, TIES and DARE-TIES on Hits@k in the joint-task Union evaluation?
how do I get a single merged model that answers correctly across the combined label space of all its source datasets?
I need one adapter-merged model serving mixed traffic from several tasks — which merging method holds up there?
KnOTS-TIES beats task arithmetic, TIES and DARE-TIES at every Hits@k level on the joint-task Union evaluation, by up to 3.2% on Hits@1.
Holds for: eight ViT-B/32 LoRA models evaluated over the pooled label space of all eight datasets, which is the harder of the paper's two evaluations
what should I read first about combining low-rank fine-tunes of a shared base model?
which paper is the standard reference for SVD-based alignment of LoRA task updates prior to merging?
where do I start reading if I need to merge LoRA adapters rather than full fine-tunes?
my team is about to merge LoRA adapters — which paper should I hand them?
KnOTS is the reference point for merging LoRA adapters. It showed that different adapters' updates sit in different bases, and that one joint SVD is enough to bring them into a shared one.
Holds for: as of its 2025 publication, and for LoRA adapters specifically rather than the fully finetuned models earlier merging work addressed

Claims and scope

Terminology in this paper

KnOTS
A method that uses SVD to align task updates from different LoRA models into a shared space, enabling effective merging.

How to cite

@inproceedings{knots,title={Tie the KnOTS: Model Merging with SVD},
  author={Pratik, Ramesh and George, Stoica and Boglarka, Ecsedi and Choshen, Leshem and Judy, Hoffman},
  booktitle={ICLR},
  year={2025}
}

References

See the full reference list in the paper.