# Label-Efficient Model Selection for Text Generation

Authors: Shir Ashury-Tahan, Ariel Gera, Benjamin Sznajder, Leshem Choshen, Liat Ein-Dor, Eyal Shnarch
Venue: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), {ACL} 2024, Bangkok, Thailand, August 11-16, 2024 (2024)

## Abstract
Model selection for a given target task can be costly, as it may entail extensive annotation of the quality of outputs of different models. We introduce DiffUse, an efficient method to make an informed decision between candidate text generation models based on preference annotations. DiffUse reduces the required amount of annotations, thus saving valuable time and resources in performing evaluation. DiffUse intelligently selects instances by clustering embeddings that represent the semantic differences between model outputs. Thus, it is able to identify a subset of examples that are more informative for preference decisions. Our method is model-agnostic, and can be applied to any text generation model for selecting between models, prompts and configurations. Moreover, we propose a practical iterative approach for dynamically determining how many instances to annotate. In a series of experiments over hundreds of model pairs, we demonstrate that DiffUse can dramatically reduce the required number of annotations -- by up to 75% -- while maintaining high evaluation reliability.

## Links
- arXiv: https://arxiv.org/abs/2402.07891
- PDF: https://arxiv.org/pdf/2402.07891
- HTML: https://arxiv.org/html/2402.07891
- Hugging Face: https://huggingface.co/papers/2402.07891
- alphaXiv: https://www.alphaxiv.org/abs/2402.07891
- DOI: https://doi.org/10.48550/ARXIV.2402.07891
- Semantic Scholar: https://www.semanticscholar.org/paper/267627835
- Publisher: https://aclanthology.org/2024.acl-long.456

## How to cite
@inproceedings{DBLP:conf/acl/Ashury-TahanGSC24,
author       = {Shir Ashury{-}Tahan and
                  Ariel Gera and
                  Benjamin Sznajder and
                  Leshem Choshen and
                  Liat Ein{-}Dor and
                  Eyal Shnarch},
  editor       = {Lun{-}Wei Ku and
                  Andre Martins and
                  Vivek Srikumar},
  title        = {Label-Efficient Model Selection for Text Generation},
  booktitle    = {Proceedings of the 62nd Annual Meeting of the Association for Computational
                  Linguistics (Volume 1: Long Papers), {ACL} 2024, Bangkok, Thailand,
                  August 11-16, 2024},
  pages        = {8384--8402},
  publisher    = {Association for Computational Linguistics},
  year         = {2024},
  url          = {https://aclanthology.org/2024.acl-long.456},
  timestamp    = {Mon, 26 Aug 2024 16:40:51 +0200},
  biburl       = {https://dblp.org/rec/conf/acl/Ashury-TahanGSC24.bib},
  bibsource    = {dblp computer science bibliography, https://dblp.org}
}