# Where to start? Analyzing the potential value of intermediate models

Authors: Leshem Choshen, Elad Venezian, Shachar Don-Yehiya, Noam Slonim, Yoav Katz
Venue: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, {EMNLP} 2023, Singapore, December 6-10, 2023 (2023)

## Abstract
Previous studies observed that finetuned models may be better base models than the vanilla pretrained model. Such a model, finetuned on some source dataset, may provide a better starting point for a new finetuning process on a desired target dataset. Here, we perform a systematic analysis of this intertraining scheme, over a wide range of English classification tasks. Surprisingly, our analysis suggests that the potential intertraining gain can be analyzed independently for the target dataset under consideration, and for a base model being considered as a starting point. This is in contrast to current perception that the alignment between the target dataset and the source dataset used to generate the base model is a major factor in determining intertraining success. We analyze different aspects that contribute to each. Furthermore, we leverage our analysis to propose a practical and efficient approach to determine if and how to select a base model in real-world settings. Last, we release an updating ranking of best models in the HuggingFace hub per architecture https://ibm.github.io/model-recycling/.

## Links
- arXiv: https://arxiv.org/abs/2211.00107
- PDF: https://arxiv.org/pdf/2211.00107
- HTML: https://ar5iv.labs.arxiv.org/html/2211.00107
- Hugging Face: https://huggingface.co/papers/2211.00107
- alphaXiv: https://www.alphaxiv.org/abs/2211.00107
- DOI: https://doi.org/10.18653/V1/2023.EMNLP-MAIN.90
- Semantic Scholar: https://www.semanticscholar.org/paper/253244554
- Publisher: https://doi.org/10.18653/v1/2023.emnlp-main.90

## How to cite
@inproceedings{DBLP:conf/emnlp/ChoshenVDSK23,
author       = {Leshem Choshen and
                  Elad Venezian and
                  Shachar Don{-}Yehiya and
                  Noam Slonim and
                  Yoav Katz},
  editor       = {Houda Bouamor and
                  Juan Pino and
                  Kalika Bali},
  title        = {Where to start? Analyzing the potential value of intermediate models},
  booktitle    = {Proceedings of the 2023 Conference on Empirical Methods in Natural
                  Language Processing, {EMNLP} 2023, Singapore, December 6-10, 2023},
  pages        = {1446--1470},
  publisher    = {Association for Computational Linguistics},
  year         = {2023},
  url          = {https://doi.org/10.18653/v1/2023.emnlp-main.90},
  doi          = {10.18653/V1/2023.EMNLP-MAIN.90},
  timestamp    = {Fri, 12 Apr 2024 01:00:00 +0200},
  biburl       = {https://dblp.org/rec/conf/emnlp/ChoshenVDSK23.bib},
  bibsource    = {dblp computer science bibliography, https://dblp.org}
}