The increasing versatility of language models (LMs) has given rise to a new class of benchmarks that comprehensively assess a broad range of capabilities. Such benchmarks are associated with massive computational costs, extending to thousands of GPU hours per model. However, the efficiency aspect of these evaluation efforts had raised little discussion in the literature.In this work, we present the problem of Efficient Benchmarking, namely, intelligently reducing the computation costs of LM evaluation without compromising reliability. Using the HELM benchmark as a test case, we investigate how different benchmark design choices affect the computation-reliability trade-off. We propose to evaluate the reliability of such decisions, by using a new measure – Decision Impact on Reliability, DIoR for short.We find, for example, that a benchmark leader may change by merely removing a low-ranked model from the benchmark, and observe that a correct benchmark ranking can be obtained by considering only a fraction of the evaluation examples.Based on our findings, we outline a set of concrete recommendations for efficient benchmark design and utilization practices. To take a step further, we use our findings to propose an evaluation algorithm, that, when applied to the HELM benchmark, leads to dramatic cost savings with minimal loss of benchmark reliability, often reducing computation by x100 or more.
@inproceedings{DBLP:conf/naacl/PerlitzBGAESSSC24,
author = {Yotam Perlitz and
Elron Bandel and
Ariel Gera and
Ofir Arviv and
Liat Ein{-}Dor and
Eyal Shnarch and
Noam Slonim and
Michal Shmueli{-}Scheuer and
Leshem Choshen},
editor = {Kevin Duh and
Helena G{\'{o}}mez{-}Adorno and
Steven Bethard},
title = {Efficient Benchmarking (of Language Models)},
booktitle = {Proceedings of the 2024 Conference of the North American Chapter of
the Association for Computational Linguistics: Human Language Technologies
(Volume 1: Long Papers), {NAACL} 2024, Mexico City, Mexico, June 16-21,
2024},
pages = {2519--2536},
publisher = {Association for Computational Linguistics},
year = {2024},
url = {https://doi.org/10.18653/v1/2024.naacl-long.139},
doi = {10.18653/V1/2024.NAACL-LONG.139},
timestamp = {Thu, 29 Aug 2024 17:13:57 +0200},
biburl = {https://dblp.org/rec/conf/naacl/PerlitzBGAESSSC24.bib},
bibsource = {dblp computer science bibliography, https://dblp.org}
}
See the full reference list in the paper.