# Efficient Benchmarking (of Language Models)

Authors: Yotam Perlitz, Elron Bandel, Ariel Gera, Ofir Arviv, Liat Ein-Dor, Eyal Shnarch, Noam Slonim, Michal Shmueli-Scheuer, Leshem Choshen
Venue: Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), {NAACL} 2024, Mexico City, Mexico, June 16-21, 2024 (2024)

## Abstract
The increasing versatility of language models (LMs) has given rise to a new class of benchmarks that comprehensively assess a broad range of capabilities. Such benchmarks are associated with massive computational costs, extending to thousands of GPU hours per model. However, the efficiency aspect of these evaluation efforts had raised little discussion in the literature.In this work, we present the problem of Efficient Benchmarking, namely, intelligently reducing the computation costs of LM evaluation without compromising reliability. Using the HELM benchmark as a test case, we investigate how different benchmark design choices affect the computation-reliability trade-off. We propose to evaluate the reliability of such decisions, by using a new measure – Decision Impact on Reliability, DIoR for short.We find, for example, that a benchmark leader may change by merely removing a low-ranked model from the benchmark, and observe that a correct benchmark ranking can be obtained by considering only a fraction of the evaluation examples.Based on our findings, we outline a set of concrete recommendations for efficient benchmark design and utilization practices. To take a step further, we use our findings to propose an evaluation algorithm, that, when applied to the HELM benchmark, leads to dramatic cost savings with minimal loss of benchmark reliability, often reducing computation by x100 or more.

## Links
- arXiv: https://arxiv.org/abs/2308.11696
- PDF: https://arxiv.org/pdf/2308.11696
- HTML: https://arxiv.org/html/2308.11696
- Hugging Face: https://huggingface.co/papers/2308.11696
- alphaXiv: https://www.alphaxiv.org/abs/2308.11696
- DOI: https://doi.org/10.18653/V1/2024.NAACL-LONG.139
- ACL Anthology: https://aclanthology.org/2024.naacl-long.139/
- Semantic Scholar: https://www.semanticscholar.org/paper/261076362
- Publisher: https://doi.org/10.18653/v1/2024.naacl-long.139

## How to cite
@inproceedings{DBLP:conf/naacl/PerlitzBGAESSSC24,
author       = {Yotam Perlitz and
                  Elron Bandel and
                  Ariel Gera and
                  Ofir Arviv and
                  Liat Ein{-}Dor and
                  Eyal Shnarch and
                  Noam Slonim and
                  Michal Shmueli{-}Scheuer and
                  Leshem Choshen},
  editor       = {Kevin Duh and
                  Helena G{\'{o}}mez{-}Adorno and
                  Steven Bethard},
  title        = {Efficient Benchmarking (of Language Models)},
  booktitle    = {Proceedings of the 2024 Conference of the North American Chapter of
                  the Association for Computational Linguistics: Human Language Technologies
                  (Volume 1: Long Papers), {NAACL} 2024, Mexico City, Mexico, June 16-21,
                  2024},
  pages        = {2519--2536},
  publisher    = {Association for Computational Linguistics},
  year         = {2024},
  url          = {https://doi.org/10.18653/v1/2024.naacl-long.139},
  doi          = {10.18653/V1/2024.NAACL-LONG.139},
  timestamp    = {Thu, 29 Aug 2024 17:13:57 +0200},
  biburl       = {https://dblp.org/rec/conf/naacl/PerlitzBGAESSSC24.bib},
  bibsource    = {dblp computer science bibliography, https://dblp.org}
}