Resolving Interference (RI): Disentangling Models for Improved Model Merging

a pre-merging adaptation step that uses unlabeled auxiliary images to make each expert model's update leave the other tasks' outputs unchanged

Pratik Ramesh, George Stoica, Arun Iyer, Leshem Choshen, Judy Hoffman · arXiv · 2026

In one sentence

Resolving Interference (RI) adapts each expert model before merging so that, on unlabeled auxiliary images, it reproduces its own task head's outputs while leaving every other task head's outputs at their pretrained values — raising the accuracy of existing merging methods without any task data.

Abstract

Model merging has shown that multitask models can be created by directly combining the parameters of different models that are each specialized on tasks of interest. However, models trained independently on distinct tasks often exhibit interference that degrades the merged model's performance. To solve this problem, we formally define the notion of Cross-Task Interference as the drift in the representation of the merged model relative to its constituent models. Reducing cross-task interference is key to improving merging performance. To address this issue, we propose our method, Resolving Interference (RI), a light-weight adaptation framework which disentangles expert models to be functionally orthogonal to the space of other tasks, thereby reducing cross-task interference. RI does this whilst using only unlabeled auxiliary data as input (i.e., no task-data is needed), allowing it to be applied in data-scarce scenarios. RI consistently improves the performance of state-of-the-art merging methods by up to 3.8% and generalization to unseen domains by up to 2.3%. We also find RI to be robust to the source of auxiliary input while being significantly less sensitive to tuning of merging hyperparameters. Our codebase is available at: https://github.com/pramesh39/resolving_interference

Questions this paper answers

can fine-tuned models be combined better when none of the original training data is available?
is there a pre-merge adaptation step that reduces cross-task interference using only unlabeled task-agnostic images?
how do I cut conflicts between expert checkpoints before merging them without touching any task's training or validation set?
I only have the expert weights and some random unlabeled images -- can I still improve my merge?
Resolving Interference reduces merging interference using only unlabeled, task-agnostic auxiliary images, with no access to any constituent task's training or validation data. Gradient-based merging adaptations such as AdaMerging or Fisher-weighted averaging instead assume access to task data.
Holds for: Positioning claim about the data-scarce setting; routing-based and task-data-based adaptation methods are excluded from comparison by design rather than shown to be worse.
Resolving Interference (RI) is an adaptation step applied to expert models before merging, not a merging method itself. RI therefore composes with existing techniques such as Task Arithmetic, TIES, KnOTS, WUDI, Iso-C, Iso-CTS and TSV-M.
Holds for: Requires expert models that share a backbone architecture and pretrained initialization, plus each task's head; demonstrated on CLIP ViT vision classifiers as of the 2026 preprint.
what does it actually mean for two combined models to conflict with each other?
how is cross-task interference between expert models and a merged model formally defined and quantified?
how do I measure how much a merged model has drifted from each of the experts it was built from?
where should I start reading if I want a precise definition of interference in model merging rather than a hand-wavy one?
Resolving Interference defines cross-task interference as the summed distance between each expert model's output representations and the merged model's output representations under that expert's own task head. An interference value of 0 is sufficient for the merged model to match every constituent expert.
Holds for: Classification settings where the distance is KL-divergence between predicted distributions; computing the metric requires task data even though the RI method itself does not, and the sufficiency statement concerns representation equality under each task head.
Resolving Interference (RI) is an adaptation step applied to expert models before merging, not a merging method itself. RI therefore composes with existing techniques such as Task Arithmetic, TIES, KnOTS, WUDI, Iso-C, Iso-CTS and TSV-M.
Holds for: Requires expert models that share a backbone architecture and pretrained initialization, plus each task's head; demonstrated on CLIP ViT vision classifiers as of the 2026 preprint.
how much extra accuracy does adapting experts first add when combining many image classifiers?
what average accuracy gain does pre-merge interference resolution give Task Arithmetic, TIES and TSV-M on the 20-task ViT-B/32 benchmark?
how much can I expect to gain by preparing each expert before merging instead of merging the checkpoints as they are?
is a pre-merge adaptation step worth adding on top of the best merging method I already use?
Adding RI before merging raises 20-task average accuracy on ViT-B/32 by 8.6 points for Task Arithmetic (56.1% to 64.7%) and 9.7 points for TIES (58.0% to 67.7%). The state-of-the-art TSV-M baseline gains 3.8 points, from 76.5% to 80.3%.
Holds for: CLIP ViT-B/32 experts from Wang et al. (2024c) on the 20-task vision benchmark, using each baseline's recommended default merging hyperparameters and ImageNet as auxiliary data.
Gains from RI grow with the number of merged tasks: TSV-M improves by 1.5, 2.8 and 3.9 points on the 8-, 14- and 20-task ViT-B/32 settings. Iso-CTS improves by 0.2, 0.9 and 1.6 points across the same three settings.
Holds for: CLIP ViT-B/32 experts, default merging hyperparameters; the trend is reported for these two merging methods across the three benchmark sizes.
does adapting models before combining them help more when there are lots of models or just a couple?
do gains from resolving cross-task interference scale with the number of merged task vectors from 8 to 20 tasks?
I need to merge 20 experts rather than 8 -- does interference reduction matter more at that scale?
my merge only has a handful of tasks in it; should I bother with pre-merge adaptation?
Gains from RI grow with the number of merged tasks: TSV-M improves by 1.5, 2.8 and 3.9 points on the 8-, 14- and 20-task ViT-B/32 settings. Iso-CTS improves by 0.2, 0.9 and 1.6 points across the same three settings.
Holds for: CLIP ViT-B/32 experts, default merging hyperparameters; the trend is reported for these two merging methods across the three benchmark sizes.
is there any way of combining models that gets no benefit from adapting the experts first?
why does simple weight averaging fail to gain from RI-adapted task vectors, and what does the scaling coefficient sweep show?
if I merge by plain weight averaging, do I need to retune the merging coefficient after adapting the experts?
I average my checkpoints rather than using task arithmetic -- will pre-merge interference reduction do anything for me?
RI improves every merging method tested on the 8/14/20-task vision benchmarks except plain weight averaging. Averaging changes by at most 1.0 point and is sometimes slightly worse with RI (ViT-B/16, 8 tasks: 72.3% without versus 71.3% with).
Holds for: ViT-B/32, ViT-B/16 and ViT-L/14 CLIP experts with default merging hyperparameters; averaging's shortfall is attributed to its very small effective scaling coefficient.
Averaging+RI in the 20-task ViT-B/32 setting rises from 61.4% at the default averaging coefficient of 0.05 to a peak of 64.7% at 0.15. Averaging therefore underuses RI's adapted task vectors rather than RI failing.
Holds for: One sweep of the scaling coefficient from 0.0 to 0.3, ViT-B/32, 20 tasks; accuracy degrades again beyond 0.15.
do models combined this way still work on image styles nobody trained on?
how does pre-merge interference resolution affect merged-model accuracy on held-out DomainNet domains?
how do I get a merged classifier that transfers to domains outside every expert's training split?
will a merged model beat the individual domain experts on image styles none of them saw?
On DomainNet, merging with RI beforehand improves mean accuracy over 5 unseen domains by 2.3 points for TIES and 2.0 points for weight averaging. The best RI-merged models reach up to 3.9 points above the split-specific expert models.
Holds for: Two CLIP ViT-B/32 experts finetuned on the real domain over disjoint halves of DomainNet's 345 classes; Iso-C and Iso-CTS show no mean gain from RI in this setting.
how can training on unrelated unlabeled images fix behaviour on the real tasks?
does minimising the interference objective on auxiliary images reduce measured cross-task interference on the in-task distributions, and how many optimisation steps does it take?
how many optimisation steps should I run the pre-merge adaptation for before the interference stops dropping?
can I trust an adaptation loss computed on out-of-task images as a proxy for my actual tasks?
Reducing the RI loss on unlabeled auxiliary images produces a corresponding drop in cross-task interference measured on the actual task data, with most of the decrease within the first 1000 optimization steps. Running RI for 25,000 steps instead of the default 2500 gains TSV-M a further 1.8 points, reaching 82.1%.
Holds for: 20-task ViT-B/32 setting with ImageNet auxiliary images, optimized for up to 25,000 steps; the correspondence is empirical, since auxiliary-data constraints do not guarantee the task-data solution.
what sort of unlabeled images do you need for the pre-merge step, and does the choice matter much?
how sensitive is interference resolution to the auxiliary distribution, from Gaussian noise and synthetic shapes to ImageNet, MSCOCO and OpenImages?
which unlabeled image source should I feed the pre-merge adaptation if I cannot use in-task data?
I have no suitable image corpus lying around -- would random noise or synthetic shapes be enough?
RI helps with auxiliary sources ranging from Gaussian noise (+0.4 points) through Shapes21k (+1.1) to ImageNet (+1.5), MSCOCO (+1.8) and OpenImages (+1.8), with in-task data as an oracle giving +3.3.
Holds for: 8-task vision setting with TSV-M as the merging method on ViT-B/32; visually diverse sources with edges and curves work better, and closer-to-task distributions work best.
how long does the extra preparation step take per model, and how much GPU memory?
what is the per-expert wall-clock and peak GPU memory cost of interference resolution at 2500 steps for ViT-B/32?
how do I budget compute for adapting 20 experts before merging them?
can I afford to run the pre-merge adaptation on a single GPU for every expert I have?
Adapting one ViT-B/32 expert with RI takes 7m07s in the 8-task setting and 8m50s in the 20-task setting at 2500 steps. Peak GPU memory stays constant at 4.8 GB on an NVIDIA A40.
Holds for: ViT-B/32 backbone, batch size 128, 2500 steps per expert, experts adapted independently and in parallel; ViT-L/14 needs a reduced batch size of 32.
which way of measuring the difference between two models' outputs works best for the pre-merge step?
does KL-divergence outperform cross-entropy and MSE as the distance in the interference-resolution loss?
which output-distance loss should I pick when aligning an expert's predictions with the merged model?
is it worth switching my distillation objective from MSE to KL for pre-merge adaptation?
KL-divergence is the most effective distance for the RI loss, lifting the average of TSV-M, Iso-C and Iso-CTS from 76.4% to 79.0%, versus 78.2% for cross-entropy and 77.7% for MSE.
Holds for: 20-task ViT-B/32 setting; all three distances improve on no adaptation, so the choice shifts the size of the gain rather than its sign.
do you still have to carefully tune the mixing strength when combining models this way?
how does resolving cross-task interference change sensitivity to merging hyperparameters, and can those hyperparameters be tuned on unlabeled auxiliary data?
how do I pick a merging scaling coefficient when I have no labeled validation data for the tasks?
can I just use the default merging coefficient instead of sweeping it on task data?
Experts adapted with RI are far less sensitive to merging-hyperparameter tuning: across 6 merging methods the mean gap between default and task-data-tuned hyperparameters is 0.4 points with RI versus 1.8 points without.
Holds for: 20-task ViT-B/32 setting, hyperparameters tuned on privileged labeled task validation data over the sweeps in Section A.2.2.
Tuning merging hyperparameters with the cross-task interference objective on unlabeled auxiliary data is worse than using defaults. The 6-method mean drops from 68.3% to 63.8% without RI and from 73.1% to 71.1% with RI.
Holds for: 20-task ViT-B/32 setting with ImageNet auxiliary data; a negative result for hyperparameter selection only, not for the RI adaptation objective itself.
is it better to fix each model before combining them or to fix the combined model afterwards?
does per-expert pre-merge interference resolution beat distilling the already-merged multitask model on the same auxiliary data?
given the same unlabeled images, should I spend them adapting each expert or distilling into the merged model?
I already have a merged checkpoint -- can I just distill it on auxiliary data rather than redoing the experts?
Adapting each expert before merging beats distilling the already-merged model on the same auxiliary data. Merge+Distill_Aux reaches 75.0% for Iso-C and 77.1% for TSV-M on 20 tasks, against 77.5% and 80.3% for RI.
Holds for: ViT-B/32, 20 tasks, ImageNet auxiliary data; Merge+Distill_Aux is at or near the unadapted baselines of 75.1% and 76.5%.

Claims and scope

Common misreadings

Terminology in this paper

Cross-task interference (ξ)
The summed expected distance between an expert model's output representations and a merged model's output representations, evaluated under that expert's own task head on that task's data.
RI loss
The sum of a task-preservation term, matching an adapted task vector's outputs to the original expert's under its own head, and an interference-reduction term, matching its outputs under every other task's head to the pretrained model's, weighted by α/(N−1).
Task-preservation objective
The requirement that an adapted task vector reproduce its original expert's outputs under its own task head.
Interference-reduction objective
The requirement that an adapted task vector leave the outputs of all other tasks' heads equal to those of the pretrained backbone, making it functionally orthogonal to those tasks.
Auxiliary data
Unlabeled, task-agnostic inputs — for example ImageNet, MSCOCO, Shapes21k or Gaussian noise — used in place of any constituent task's data during adaptation.
Merge+Distill_Aux
The alternative baseline of merging first and then distilling the single merged model toward each expert on auxiliary data, rather than adapting each expert before merging.

How to cite

@misc{ramesh2026resolving,
  title={Resolving Interference (RI): Disentangling Models for Improved Model Merging},
  author={Ramesh, Pratik and Stoica, George and Iyer, Arun and Choshen, Leshem and Hoffman, Judy},
  year={2026},
  url       = {https://arxiv.org/abs/2603.13467},
  doi       = {10.48550/ARXIV.2603.13467},
  publisher = {arXiv}
}

References

See the full reference list in the paper.