A Survey on Model MoErging: Recycling and Routing Among Specialized Experts for Collaborative Learning

recycling independently fine-tuned expert models by learning a router that picks and combines them per query or per task

Prateek Yadav, Colin Raffel, Mohammed Muqeeth, Lucas Caccia, Haokun Liu, Tianlong Chen, Mohit Bansal, Leshem Choshen, Alessandro Sordoni · TMLR · 2025

In one sentence

A survey of model MoErging — recycling independently trained expert models via a learned router — that catalogues a few dozen methods along a 9-axis taxonomy of expert, routing and application design choices, and groups them into embedding-based, classifier-based, task-specific and router-free routing.

Abstract

The availability of performant pre-trained models has led to a proliferation of fine-tuned expert models that are specialized to a particular domain or task. Model MoErging methods aim to recycle expert models to create an aggregate system with improved performance or generalization. A key component of MoErging methods is the creation of a router that decides which expert model(s) to use for a particular input or application. The promise, effectiveness, and large design space of MoErging has spurred the development of many new methods over the past few years. This rapid pace of development has made it challenging to compare different MoErging methods, which are rarely compared to one another and are often validated in different experimental setups. To remedy such gaps, we present a comprehensive survey of MoErging methods that includes a novel taxonomy for cataloging key design choices and clarifying suitable applications for each method. Apart from surveying MoErging research, we inventory software tools and applications that make use of MoErging. We additionally discuss related fields of study such as model merging, multitask learning, and mixture-of-experts models. Taken as a whole, our survey provides a unified overview of existing MoErging methods and creates a solid foundation for future work in this burgeoning field.

Questions this paper answers

Is there a survey of methods that reuse many separately fine-tuned models by picking between them automatically?
What survey organises post-hoc routing over independently trained expert modules such as LoRA adapters?
Where do I start reading about combining a pool of community fine-tuned adapters with a learned router?
I have dozens of LoRA adapters lying around, is there a review that tells me what my options are for serving them together?
The MoErging survey introduces a taxonomy that catalogues each surveyed method along 9 design choices. The axes are expert training procedure and expert data privacy, then routing dataset, input granularity, depth granularity, expert selection and expert aggregation, then generalization target and user dataset requirement.
Holds for: Covers methods surveyed as of the 2024 arXiv release and 2025 TMLR publication; some papers map onto an axis only as 'Multiple' or 'N/A', and the authors state the taxonomy may miss methods and design choices.
MoErging methods fall into 4 families by how the router is built: embedding-based routing, classifier-based routing, task-specific routing, and router-free approaches. The remaining differences in granularity, selection and aggregation are comparatively superficial.
Holds for: A grouping the survey argues for over the few dozen methods it reviews in Section 4; methods that mix strategies fit the grouping less cleanly.
MoErging learns post-hoc routing over independently trained, decentralized experts and combines them adaptively per query or per task. That separates it from mixture-of-experts models, which jointly train experts and router from scratch, and from model merging, which produces a static combination.
Holds for: The survey's own delineation; the authors note that precisely delineating what is and is not MoErging is challenging because adjacent methods share the motivation while differing in framing.
How is picking between separately trained expert models different from a mixture-of-experts network or from averaging model weights?
What separates post-hoc routing over decentralized experts from jointly trained sparse MoE and from static parameter merging?
How do I decide between merging my fine-tuned checkpoints into one model and routing between them at query time?
Should I treat a router over my existing adapters as an MoE model, or is it a different kind of system?
MoErging learns post-hoc routing over independently trained, decentralized experts and combines them adaptively per query or per task. That separates it from mixture-of-experts models, which jointly train experts and router from scratch, and from model merging, which produces a static combination.
Holds for: The survey's own delineation; the authors note that precisely delineating what is and is not MoErging is challenging because adjacent methods share the motivation while differing in framing.
What are the main ways of deciding which fine-tuned model to use for a given input?
How can post-hoc routing mechanisms over pre-trained expert modules be grouped into families?
How do I choose an approach for building a router over a library of fine-tuned experts?
Which style of expert selection should I build for my adapter pool, similarity search, a trained classifier, or something without a router?
MoErging methods fall into 4 families by how the router is built: embedding-based routing, classifier-based routing, task-specific routing, and router-free approaches. The remaining differences in granularity, selection and aggregation are comparatively superficial.
Holds for: A grouping the survey argues for over the few dozen methods it reviews in Section 4; methods that mix strategies fit the grouping less cleanly.
Embedding-based routers, the 7 of which the survey lists include AdapterSoup, Retrieval of Experts, LoraRetriever and Dynamic Adapter Merging, need little or no routing-specific training data and tolerate experts with different architectures. Their quality is bounded by the pre-trained embedding space and degrades on semantically subtle, highly specialized, or out-of-distribution tasks.
Holds for: The embedding-based family as characterised in the survey's grouping; assumes expert training distributions are distinct enough to separate in embedding space.
Classifier-based routers, the 5 of which the survey lists are Zooter, RouteLLM, Routoo, Branch-Train-Mix and Routing with Benchmark Datasets, can learn routing functions more complex than embedding similarity. They depend on labeled routing data and therefore lose applicability in zero-shot and few-shot settings.
Holds for: The classifier-based family; the labeled data may come from expert data, a general dataset, or the target task, and the constraint is data availability rather than accuracy.
Router-free approaches such as Arrow, PHATGOOSE, Airoboros and LlamaIndex avoid post-hoc router training entirely, using precomputed gates, LoRA prototypes or a pre-trained LLM's own knowledge. That gives zero-shot deployment with no routing data, at the cost of possibly modified expert training and less stable routing.
Holds for: The router-free family in the survey's grouping; PHATGOOSE and Arrow require expert-side changes or precomputation, so 'no training' refers to the router, not the experts.
Which ways of choosing between fine-tuned models work without collecting any labelled examples of which model to use?
Which expert-routing approaches support zero-shot deployment with no routing-specific training data?
How do I route among LoRA experts when I have no labelled routing data and cannot train a router?
Can I serve a pool of adapters with automatic selection without first labelling data for the routing decision?
Router-free approaches such as Arrow, PHATGOOSE, Airoboros and LlamaIndex avoid post-hoc router training entirely, using precomputed gates, LoRA prototypes or a pre-trained LLM's own knowledge. That gives zero-shot deployment with no routing data, at the cost of possibly modified expert training and less stable routing.
Holds for: The router-free family in the survey's grouping; PHATGOOSE and Arrow require expert-side changes or precomputation, so 'no training' refers to the router, not the experts.
Embedding-based routers, the 7 of which the survey lists include AdapterSoup, Retrieval of Experts, LoraRetriever and Dynamic Adapter Merging, need little or no routing-specific training data and tolerate experts with different architectures. Their quality is bounded by the pre-trained embedding space and degrades on semantically subtle, highly specialized, or out-of-distribution tasks.
Holds for: The embedding-based family as characterised in the survey's grouping; assumes expert training distributions are distinct enough to separate in embedding space.
When is training a small model to pick the right expert better than just comparing text embeddings?
What are the tradeoffs of classifier-based routing versus embedding-similarity routing for expert selection?
How do I choose between an embedding-similarity router and a trained classifier for dispatching queries to LLM experts?
Do I need to collect labelled routing data if I want a trained router for picking between my models?
Classifier-based routers, the 5 of which the survey lists are Zooter, RouteLLM, Routoo, Branch-Train-Mix and Routing with Benchmark Datasets, can learn routing functions more complex than embedding similarity. They depend on labeled routing data and therefore lose applicability in zero-shot and few-shot settings.
Holds for: The classifier-based family; the labeled data may come from expert data, a general dataset, or the target task, and the constraint is data availability rather than accuracy.
Embedding-based routers, the 7 of which the survey lists include AdapterSoup, Retrieval of Experts, LoraRetriever and Dynamic Adapter Merging, need little or no routing-specific training data and tolerate experts with different architectures. Their quality is bounded by the pre-trained embedding space and degrades on semantically subtle, highly specialized, or out-of-distribution tasks.
Holds for: The embedding-based family as characterised in the survey's grouping; assumes expert training distributions are distinct enough to separate in embedding space.
Should the choice of which fine-tuned model to use be made once per task, per input, or at every generation step?
How does routing granularity, per-task versus per-example versus per-step, trade adaptability against cost and stability?
How do I pick the granularity of routing decisions when serving a mixture over adapters?
Is per-token routing over my adapters worth the extra compute compared with deciding once per request?
Routing granularity trades adaptability against cost and stability across the 3 levels the MoErging taxonomy defines: per-task, per-example and per-step. Per-step routing adapts most to the input but is expensive and risks early routing errors propagating, while per-task routing is cheapest and assumes constant expert needs within a task.
Holds for: A synthesis of trade-offs across the surveyed methods rather than a controlled experiment; the same error-propagation concern is raised for per-module routing depth.
Why is it hard to tell which method for picking between fine-tuned models is actually better?
How consistently do post-hoc expert-routing papers evaluate against each other and under matched data-access assumptions?
How do I compare published expert-routing methods when I need to choose one to implement?
Can I trust reported numbers to tell me which adapter-routing method to use?
MoErging papers rarely compare to one another: among the 7 embedding-based routing methods the survey lists, only LoraRetriever compares against AdapterSoup. Most surveyed papers cite only a small fraction of the others.
Holds for: Based on the papers catalogued in Section 4 as of the survey's writing; the observation is about published comparisons, not about whether the methods are comparable in principle.
Data-access assumptions decide which MoErging methods are usable at all: a method requiring a labeled target-task dataset cannot by definition improve zero-shot generalization. A method requiring shared expert training data cannot be applied to publicly shared adapters, which are seldom released with their training data.
Holds for: A logical consequence of the survey's taxonomy axes for routing dataset, expert data and user dataset, stated over the methods catalogued in Section 4.
If someone has all the training data for every expert model, what simple alternative should their routing method be compared against?
What baseline is missing from expert-routing papers that assume simultaneous access to all expert training datasets?
What should I benchmark my router over fine-tuned experts against when I own all the expert training sets?
Before building a router over my own fine-tuned models, should I just train one multitask model instead?
When a MoErging method assumes all expert training datasets are simultaneously available, multitask training of the base model on those expert datasets is an applicable baseline that is often omitted. Multitask mixture-of-experts methods likewise become valid baselines under shared expert data.
Holds for: Methods whose taxonomy entry lists expert data as Shared; methods that keep expert data Private cannot be compared to a multitask baseline on the same terms.
What data does a method for choosing between fine-tuned models need, and when does that rule it out?
Which data-access assumptions, such as labeled target-task data or shared expert training sets, gate the applicability of expert-routing methods?
How do I tell whether an expert-routing method can be applied to adapters downloaded from a model hub?
I only have the adapter weights and no training data behind them, which routing approaches are even usable for me?
Data-access assumptions decide which MoErging methods are usable at all: a method requiring a labeled target-task dataset cannot by definition improve zero-shot generalization. A method requiring shared expert training data cannot be applied to publicly shared adapters, which are seldom released with their training data.
Holds for: A logical consequence of the survey's taxonomy axes for routing dataset, expert data and user dataset, stated over the methods catalogued in Section 4.
Why do so few of the published ways of reusing other people's fine-tuned models get used in real systems?
What blocks practical adoption of post-hoc routing over decentralized expert modules despite reported gains?
What should I expect to have to build myself if I want to deploy adaptive routing over community adapters?
Are any of these expert-routing methods mature enough for me to put in production?
Very few of the MoErging methods catalogued in the survey are actually used in practice, despite many demonstrating improved performance or generalization. The survey attributes this to missing user-friendly implementations, unclear guidance on which method suits which use case, and assumptions such as custom expert training or shared expert data.
Holds for: An assessment as of publication; the attributed causes are the authors' speculation rather than a measured result, and the tools inventory in Section 6 shows supporting infrastructure does exist.
The MoErging survey inventories the software ecosystem for decentralized expert reuse, including Hugging Face PEFT, AdapterHub, Git-theta, MergeKit, Predibase's Lorax, Mergoo, Flow-Merge, Axolotl, Unsloth, airoboros and ComfyUI. It notes that no existing platform coordinates continual communal model development.
Holds for: An inventory as of publication; the tool landscape changes quickly and the survey does not benchmark the tools against one another.
What existing software can combine or switch between many fine-tuned versions of a model?
Which libraries and platforms support merging, serving and routing over PEFT adapters and expert checkpoints?
Which tools do I use to host a mixture over many LoRA adapters?
Is there a ready-made library for serving hundreds of adapters, or do I need to write the routing layer myself?
The MoErging survey inventories the software ecosystem for decentralized expert reuse, including Hugging Face PEFT, AdapterHub, Git-theta, MergeKit, Predibase's Lorax, Mergoo, Flow-Merge, Axolotl, Unsloth, airoboros and ComfyUI. It notes that no existing platform coordinates continual communal model development.
Holds for: An inventory as of publication; the tool landscape changes quickly and the survey does not benchmark the tools against one another.
What still needs solving before communities can keep improving a shared model by contributing fine-tuned pieces?
What open problems remain in expert-pool maintenance, including redundancy removal, admission decisions and detection of malicious experts?
How do I manage a growing shared pool of contributed experts, including duplicates and bad-faith uploads?
If I open a shared adapter pool to contributors, what problems have no good answer yet?
The MoErging survey names as open problems identifying and removing redundant experts, deciding whether to admit a new expert to the pool, and detecting maliciously contributed experts. It also asks whether repeated rounds of MoErging can continually improve a base model rather than giving one-off gains.
Holds for: Directions the authors identify as of publication, alongside a call for theoretical frameworks and for merging-style benchmarks and competitions; none are addressed experimentally in the survey.
Very few of the MoErging methods catalogued in the survey are actually used in practice, despite many demonstrating improved performance or generalization. The survey attributes this to missing user-friendly implementations, unclear guidance on which method suits which use case, and assumptions such as custom expert training or shared expert data.
Holds for: An assessment as of publication; the attributed causes are the authors' speculation rather than a measured result, and the tools inventory in Section 6 shows supporting infrastructure does exist.
What questions should I ask about a method that picks between fine-tuned models before deciding if it fits my setup?
Along which design axes can expert-routing methods be characterised, from expert training and data privacy to routing granularity and generalization target?
How do I compare two adapter-routing methods on their assumptions about expert data and user-provided data?
How do I check whether an expert-routing paper's assumptions match the data I actually have?
The MoErging survey introduces a taxonomy that catalogues each surveyed method along 9 design choices. The axes are expert training procedure and expert data privacy, then routing dataset, input granularity, depth granularity, expert selection and expert aggregation, then generalization target and user dataset requirement.
Holds for: Covers methods surveyed as of the 2024 arXiv release and 2025 TMLR publication; some papers map onto an axis only as 'Multiple' or 'N/A', and the authors state the taxonomy may miss methods and design choices.
Data-access assumptions decide which MoErging methods are usable at all: a method requiring a labeled target-task dataset cannot by definition improve zero-shot generalization. A method requiring shared expert training data cannot be applied to publicly shared adapters, which are seldom released with their training data.
Holds for: A logical consequence of the survey's taxonomy axes for routing dataset, expert data and user dataset, stated over the methods catalogued in Section 4.

Claims and scope

Common misreadings

Terminology in this paper

MoErging
A paradigm in which expert models fine-tuned independently by distributed contributors are recycled into an aggregate system by learning a post-hoc router that adaptively selects and combines them per query or per task.
Expert data (Shared vs Private)
A taxonomy axis recording whether a routing method requires contributors to release the expert models' training datasets (Shared) or can operate on the expert parameters alone (Private).
Routing dataset
The data used to learn the router, categorized as the experts' own training data, a target-task dataset, a general multi-task dataset, or none for methods that use heuristics or precomputed information instead of router training.
Routing input granularity
Whether a routing decision is made once per task, once per input example, or at every generation step or token.
Routing depth granularity
Whether one routing decision applies to the whole model or an independent decision is made at each layer or module where experts are inserted.
Expert aggregation
How multiple selected experts are combined: mixing their outputs, merging their parameter values into a single model before processing the input, or no aggregation when a single expert is selected.
Router-free approaches
Expert-selection methods that determine expert choice from precomputed gates, LoRA prototypes or a pre-trained LLM's knowledge, without training a dedicated router after the experts exist.

How to cite

@article{yadav2024survey,
title={A Survey on Model MoErging: Recycling and Routing Among Specialized Experts for Collaborative Learning},
author={Prateek Yadav and Colin Raffel and Mohammed Muqeeth and Lucas Caccia and Haokun Liu and Tianlong Chen and Mohit Bansal and Leshem Choshen and Alessandro Sordoni},
journal={Transactions on Machine Learning Research},
issn={2835-8856},
year={2025},
url={https://openreview.net/forum?id=u0azVc9Y0y},
note={Survey Certification}
}

References

See the full reference list in the paper.