Pretraining has been shown to scale well with compute, data size and data diversity. Multitask learning trains on a mixture of supervised datasets and produces improved performance compared to self-supervised pretraining.Until now, massively multitask learning required simultaneous access to all datasets in the mixture and heavy compute resources that are only available to well-resourced teams. In this paper, we propose ColD Fusion, a method that provides the benefits of multitask learning but leverages distributed computation and requires limited communication and no sharing of data. Consequentially, ColD Fusion can create a synergistic loop, where finetuned models can be recycled to continually improve the pretrained model they are based on.We show that ColD Fusion yields comparable benefits to multitask training by producing a model that (a) attains strong performance on all of the datasets it was multitask trained on and (b) is a better starting point for finetuning on unseen datasets. We find ColD Fusion outperforms RoBERTa and even previous multitask models. Specifically, when training and testing on 35 diverse datasets, ColD Fusion-based model outperforms RoBERTa by 2.19 points on average without any changes to the architecture.
@inproceedings{DBLP:conf/acl/Don-YehiyaVRSC23,
author = {Shachar Don{-}Yehiya and
Elad Venezian and
Colin Raffel and
Noam Slonim and
Leshem Choshen},
editor = {Anna Rogers and
Jordan L. Boyd{-}Graber and
Naoaki Okazaki},
title = {ColD Fusion: Collaborative Descent for Distributed Multitask Finetuning},
booktitle = {Proceedings of the 61st Annual Meeting of the Association for Computational
Linguistics (Volume 1: Long Papers), {ACL} 2023, Toronto, Canada,
July 9-14, 2023},
pages = {788--806},
publisher = {Association for Computational Linguistics},
year = {2023},
url = {https://doi.org/10.18653/v1/2023.acl-long.46},
doi = {10.18653/V1/2023.ACL-LONG.46},
timestamp = {Thu, 10 Aug 2023 01:00:00 +0200},
biburl = {https://dblp.org/rec/conf/acl/Don-YehiyaVRSC23.bib},
bibsource = {dblp computer science bibliography, https://dblp.org}
}
See the full reference list in the paper.