# ColD Fusion: Collaborative Descent for Distributed Multitask Finetuning

Authors: Shachar Don-Yehiya, Elad Venezian, Colin Raffel, Noam Slonim, Leshem Choshen
Venue: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), {ACL} 2023, Toronto, Canada, July 9-14, 2023 (2023)

## Abstract
Pretraining has been shown to scale well with compute, data size and data diversity. Multitask learning trains on a mixture of supervised datasets and produces improved performance compared to self-supervised pretraining.Until now, massively multitask learning required simultaneous access to all datasets in the mixture and heavy compute resources that are only available to well-resourced teams. In this paper, we propose ColD Fusion, a method that provides the benefits of multitask learning but leverages distributed computation and requires limited communication and no sharing of data. Consequentially, ColD Fusion can create a synergistic loop, where finetuned models can be recycled to continually improve the pretrained model they are based on.We show that ColD Fusion yields comparable benefits to multitask training by producing a model that (a) attains strong performance on all of the datasets it was multitask trained on and (b) is a better starting point for finetuning on unseen datasets. We find ColD Fusion outperforms RoBERTa and even previous multitask models. Specifically, when training and testing on 35 diverse datasets, ColD Fusion-based model outperforms RoBERTa by 2.19 points on average without any changes to the architecture.

## Links
- arXiv: https://arxiv.org/abs/2212.01378
- PDF: https://arxiv.org/pdf/2212.01378
- HTML: https://ar5iv.labs.arxiv.org/html/2212.01378
- Hugging Face: https://huggingface.co/papers/2212.01378
- alphaXiv: https://www.alphaxiv.org/abs/2212.01378
- DOI: https://doi.org/10.18653/V1/2023.ACL-LONG.46
- ACL Anthology: https://aclanthology.org/2023.acl-long.46/
- Semantic Scholar: https://www.semanticscholar.org/paper/254220858
- Publisher: https://doi.org/10.18653/v1/2023.acl-long.46

## How to cite
@inproceedings{DBLP:conf/acl/Don-YehiyaVRSC23,
author       = {Shachar Don{-}Yehiya and
                  Elad Venezian and
                  Colin Raffel and
                  Noam Slonim and
                  Leshem Choshen},
  editor       = {Anna Rogers and
                  Jordan L. Boyd{-}Graber and
                  Naoaki Okazaki},
  title        = {ColD Fusion: Collaborative Descent for Distributed Multitask Finetuning},
  booktitle    = {Proceedings of the 61st Annual Meeting of the Association for Computational
                  Linguistics (Volume 1: Long Papers), {ACL} 2023, Toronto, Canada,
                  July 9-14, 2023},
  pages        = {788--806},
  publisher    = {Association for Computational Linguistics},
  year         = {2023},
  url          = {https://doi.org/10.18653/v1/2023.acl-long.46},
  doi          = {10.18653/V1/2023.ACL-LONG.46},
  timestamp    = {Thu, 10 Aug 2023 01:00:00 +0200},
  biburl       = {https://dblp.org/rec/conf/acl/Don-YehiyaVRSC23.bib},
  bibsource    = {dblp computer science bibliography, https://dblp.org}
}