Efficient Benchmarking (of Language Models)

Yotam Perlitz; Elron Bandel; Ariel Gera; Ofir Arviv; Liat Ein-Dor; Eyal Shnarch; Noam Slonim; Michal Shmueli-Scheuer; Leshem Choshen

doi:10.18653/V1/2024.NAACL-LONG.139

Efficient Benchmarking (of Language Models)

a cheap tier-based version of the HELM benchmark that spends more compute only on models near the top of the ranking

Yotam Perlitz, Elron Bandel, Ariel Gera, Ofir Arviv, Liat Ein-Dor, Eyal Shnarch, Noam Slonim, Michal Shmueli-Scheuer, Leshem Choshen · NAACL 2024 · 2024

In one sentence

Efficient Benchmarking treats the compute cost of LM evaluation as a design variable, measures how each benchmark design choice affects ranking stability with a new Decision Impact on Reliability (DIoR) measure, and shows that on HELM most compute can be cut by dropping examples rather than datasets.

Abstract

The increasing versatility of language models (LMs) has given rise to a new class of benchmarks that comprehensively assess a broad range of capabilities. Such benchmarks are associated with massive computational costs, extending to thousands of GPU hours per model. However, the efficiency aspect of these evaluation efforts had raised little discussion in the literature.In this work, we present the problem of Efficient Benchmarking, namely, intelligently reducing the computation costs of LM evaluation without compromising reliability. Using the HELM benchmark as a test case, we investigate how different benchmark design choices affect the computation-reliability trade-off. We propose to evaluate the reliability of such decisions, by using a new measure – Decision Impact on Reliability, DIoR for short.We find, for example, that a benchmark leader may change by merely removing a low-ranked model from the benchmark, and observe that a correct benchmark ranking can be obtained by considering only a fraction of the evaluation examples.Based on our findings, we outline a set of concrete recommendations for efficient benchmark design and utilization practices. To take a step further, we use our findings to propose an evaluation algorithm, that, when applied to the HELM benchmark, leads to dramatic cost savings with minimal loss of benchmark reliability, often reducing computation by x100 or more.

Questions this paper answers

why does running a big language-model benchmark cost so much, and is anyone studying how to cut that cost?
what does efficient benchmarking of LLMs mean, and how are evaluation compute budgets analysed against reliability?
where do I start reading about making language-model evaluation cheaper without making it less trustworthy?
I cannot afford to run a full evaluation suite on every checkpoint, is there work that tells me what to cut?: Efficient Benchmarking names the problem of cutting the computation cost of language-model evaluation without losing reliability. It argues that benchmark design choices should be judged by their measured effect on reliability rather than by intuition.
Holds for: Framing introduced in 2024; the accompanying empirical study covers only the HELM benchmark, and other benchmarks and decisions such as prompt templates were left to future work.; Evaluating a single model on the HELM benchmark can cost $10K or more than 4K GPU hours, which is why the compute side of benchmark design is worth studying at all.
Holds for: Cost reported for HELM by Liang et al. (2022) and cited as of 2023-2024; shifts with hardware and inference pricing.
if I need to shrink a language-model evaluation suite, is it better to drop whole datasets or to test fewer questions from each?
in HELM, does reducing scenarios or reducing examples per scenario degrade ranking reliability more?
how do I cut the size of an evaluation suite without changing which model comes out ahead?
should I evaluate on fewer tasks or on fewer examples per task to save inference budget?: On HELM, the choice of evaluation examples is highly reliable while the choice of the 16 scenarios and 40 subscenarios supports only low reliability. Cutting compute by dropping examples is therefore sound, and dropping datasets is not.
Holds for: 37 models on HELM v0.2.2 over 16 core scenarios, 40 subscenarios and 65K examples; reliability estimated by bootstrap resampling 1K times, assuming other equally valid dataset choices exist.; HELM model ranks stay nearly identical when the number of examples per scenario is reduced 10x, and a 400x reduction still clusters models into the same small groups seen at full compute.
Holds for: 37 models on HELM v0.2.2, ranking by Mean Win Rate over 16 core scenarios; concerns rank groups rather than exact rank of any single model.
do language-model leaderboard rankings shift if only a small sample of test questions is used?
how far can HELM examples-per-scenario be sub-sampled before model rank order and rank resolution change?
how few evaluation examples per dataset can I run and still trust the ordering of models I get?
can I evaluate on a small random sample of a benchmark and still report the same model ranking?: HELM model ranks stay nearly identical when the number of examples per scenario is reduced 10x, and a 400x reduction still clusters models into the same small groups seen at full compute.
Holds for: 37 models on HELM v0.2.2, ranking by Mean Win Rate over 16 core scenarios; concerns rank groups rather than exact rank of any single model.; Distinguishing HELM models three ranks apart to an average error rate under 5% needs only 1/4 of the benchmark's computation. For clusters of 10 or 20 adjacent models a hundredth of the cost or less suffices, while clusters of size 2 switch places even with all examples.
Holds for: 37 models on HELM v0.2.2; error rate is the probability that the first and last model of a rank cluster switch places under a different random choice of examples, averaged over 1K iterations and over the top 5 models as the top model.; With a bare minimum of HELM examples models already collapse into equivalence classes of about 2-5 models. A few hundred examples reach separation into groups of roughly 2, the finest resolution the full benchmark ever achieves.
Holds for: 37 models on HELM v0.2.2; per-model rank error ranges from 6 to 2 ranks (Figure 6), so adjacent-rank distinctions remain unreliable even at full compute.
can a benchmark actually tell you which language model is the single best one?
is the top-1 model identification objective on HELM reliable, or only coarse-grained ranking and model-quality objectives?
how do I tell whether a one-rank or two-rank gap between models on a leaderboard means anything?
the model at the top of a leaderboard beat second place by a hair, should I pick it?: Identifying the single best model on HELM is unreliable even at full compute, while the full-ranking and model-quality objectives are reliable. Claims about which model is top should therefore not be drawn from HELM's bottom-line score.
Holds for: 37 models on HELM v0.2.2 with the best-model meta-metric defined as the probability of a rank switch between the top two models, repeated 5 times each time removing the current top model.; With a bare minimum of HELM examples models already collapse into equivalence classes of about 2-5 models. A few hundred examples reach separation into groups of roughly 2, the finest resolution the full benchmark ever achieves.
Holds for: 37 models on HELM v0.2.2; per-model rank error ranges from 6 to 2 ranks (Figure 6), so adjacent-rank distinctions remain unreliable even at full compute.
how can you tell whether a choice made in designing a benchmark gives trustworthy results?
what meta-metric quantifies the reliability of a benchmark design decision over rankings, and how is its confidence bound computed?
how do I score two alternative evaluation setups against each other for ranking reliability rather than by intuition?
before I change my evaluation setup, is there a number I can compute to check the rankings will still hold?: DIoR (Decision Impact on Reliability) scores a benchmark design decision by the reliability of the rankings it produces. The score is the lower bound of a bootstrap 95% confidence interval on a similarity meta-metric between rankings from alternative instantiations of the decision.
Holds for: Requires a distribution of plausible alternative instantiations of the decision, such as resampled datasets or examples, and a meta-metric such as Kendall tau.
when averaging benchmark scores, is it a mistake to first group datasets into task categories?
does aggregating HELM subscenarios into scenarios before averaging reduce ranking reliability relative to treating each subscenario standalone?
how should I aggregate per-dataset scores into one benchmark number so model comparisons stay stable?
should I report my benchmark average over task groups or over the individual datasets?: Treating each HELM subscenario as a standalone scenario cuts the error rate between top pairs of models from 22% to 14%. Aggregating the 40 subscenarios into 16 scenarios therefore costs reliability.
Holds for: 37 models on HELM v0.2.2 with Mean Win Rate aggregation; Kendall tau correlations between subscenario rankings are no higher within a scenario than across scenarios.
is it better to test every question with every prompt wording, or to pick a different wording per question?
how should few-shot prompt variation be sampled across evaluated instances to maximise ranking reliability per unit of inference compute?
how do I spread a fixed inference budget across prompt variants and test examples?
do I have to run all my prompt templates on every example, or can I sample one per example and save compute?: Sampling a different few-shot prompt for each evaluated HELM example raises reliability over running every example against all 3 prompt sets. Under the uniform scheme more than half of the compute can be saved at no cost to reliability.
Holds for: Fixed budget of 3K inference calls on HELM subscenarios that varied their in-context examples; only 3 prompt sets were available, so the effect is likely underestimated and no bootstrap over prompts was possible.
can someone move to the top of a win-rate leaderboard just by adding extra weak models to the comparison?
is HELM's Mean Win Rate aggregation sensitive to the set of models included, and can leader order flip on adding a low-ranked model?
how do I check whether my leaderboard's aggregation changes the winner when the model pool changes?
should I trust a mean-win-rate leaderboard where the set of compared models keeps growing?: HELM's Mean Win Rate aggregation lets a benchmark leader change when a low-ranked model is added. The top two models, davinci2 and Cohere XXL, switch places depending only on whether Cohere Medium is included in the comparison.
Holds for: HELM v0.2.2 scores with Mean Win Rate as the bottom-line metric; follows from the comparative nature of win-rate aggregation, and applies to any benchmark that releases several similar sizes of a model.
is there a cheap way to run a large benchmark that still puts the models in the right order?
how much compute does Flash-HELM save relative to full HELM while preserving rank resolution at each tier?
how do I place a new model against an existing leaderboard without running the whole benchmark on it?
can I rank my model against a published leaderboard for a fraction of the GPU hours?: Flash-HELM reproduces HELM ranks within the required rank resolution while reducing computation by up to 200x. It assigns each rank tier a required resolution and escalates sub-sample size from 20 through 1000 examples only as needed.
Holds for: Evaluated on the 7 models newly introduced in HELM v0.2.3; tier resolutions were fitted on the 37 models of HELM v0.2.2, and the reference is HELM's reported ranks rather than a true ranking.

Claims and scope

Efficient Benchmarking names the problem of cutting the computation cost of language-model evaluation without losing reliability. It argues that benchmark design choices should be judged by their measured effect on reliability rather than by intuition.
Scope: Framing introduced in 2024; the accompanying empirical study covers only the HELM benchmark, and other benchmarks and decisions such as prompt templates were left to future work.
DIoR (Decision Impact on Reliability) scores a benchmark design decision by the reliability of the rankings it produces. The score is the lower bound of a bootstrap 95% confidence interval on a similarity meta-metric between rankings from alternative instantiations of the decision.
Scope: Requires a distribution of plausible alternative instantiations of the decision, such as resampled datasets or examples, and a meta-metric such as Kendall tau.
On HELM, the choice of evaluation examples is highly reliable while the choice of the 16 scenarios and 40 subscenarios supports only low reliability. Cutting compute by dropping examples is therefore sound, and dropping datasets is not. (Figure 2)
Scope: 37 models on HELM v0.2.2 over 16 core scenarios, 40 subscenarios and 65K examples; reliability estimated by bootstrap resampling 1K times, assuming other equally valid dataset choices exist.
HELM model ranks stay nearly identical when the number of examples per scenario is reduced 10x, and a 400x reduction still clusters models into the same small groups seen at full compute. (Figure 1)
Scope: 37 models on HELM v0.2.2, ranking by Mean Win Rate over 16 core scenarios; concerns rank groups rather than exact rank of any single model.
With a bare minimum of HELM examples models already collapse into equivalence classes of about 2-5 models. A few hundred examples reach separation into groups of roughly 2, the finest resolution the full benchmark ever achieves. (Figure 1)
Scope: 37 models on HELM v0.2.2; per-model rank error ranges from 6 to 2 ranks (Figure 6), so adjacent-rank distinctions remain unreliable even at full compute.
Distinguishing HELM models three ranks apart to an average error rate under 5% needs only 1/4 of the benchmark's computation. For clusters of 10 or 20 adjacent models a hundredth of the cost or less suffices, while clusters of size 2 switch places even with all examples. (Figure 3)
Scope: 37 models on HELM v0.2.2; error rate is the probability that the first and last model of a rank cluster switch places under a different random choice of examples, averaged over 1K iterations and over the top 5 models as the top model.
Identifying the single best model on HELM is unreliable even at full compute, while the full-ranking and model-quality objectives are reliable. Claims about which model is top should therefore not be drawn from HELM's bottom-line score. (Figure 2)
Scope: 37 models on HELM v0.2.2 with the best-model meta-metric defined as the probability of a rank switch between the top two models, repeated 5 times each time removing the current top model.
Treating each HELM subscenario as a standalone scenario cuts the error rate between top pairs of models from 22% to 14%. Aggregating the 40 subscenarios into 16 scenarios therefore costs reliability. (Appendix F)
Scope: 37 models on HELM v0.2.2 with Mean Win Rate aggregation; Kendall tau correlations between subscenario rankings are no higher within a scenario than across scenarios.
Sampling a different few-shot prompt for each evaluated HELM example raises reliability over running every example against all 3 prompt sets. Under the uniform scheme more than half of the compute can be saved at no cost to reliability. (Figure 4)
Scope: Fixed budget of 3K inference calls on HELM subscenarios that varied their in-context examples; only 3 prompt sets were available, so the effect is likely underestimated and no bootstrap over prompts was possible.
HELM's Mean Win Rate aggregation lets a benchmark leader change when a low-ranked model is added. The top two models, davinci2 and Cohere XXL, switch places depending only on whether Cohere Medium is included in the comparison. (Section 5.5)
Scope: HELM v0.2.2 scores with Mean Win Rate as the bottom-line metric; follows from the comparative nature of win-rate aggregation, and applies to any benchmark that releases several similar sizes of a model.
Flash-HELM reproduces HELM ranks within the required rank resolution while reducing computation by up to 200x. It assigns each rank tier a required resolution and escalates sub-sample size from 20 through 1000 examples only as needed. (Figure 5)
Scope: Evaluated on the 7 models newly introduced in HELM v0.2.3; tier resolutions were fitted on the 37 models of HELM v0.2.2, and the reference is HELM's reported ranks rather than a true ranking.
Evaluating a single model on the HELM benchmark can cost $10K or more than 4K GPU hours, which is why the compute side of benchmark design is worth studying at all.
Scope: Cost reported for HELM by Liang et al. (2022) and cited as of 2023-2024; shifts with hardware and inference pricing.

Common misreadings

A 100x compute reduction on HELM preserves rank groups, not exact ranks: models adjacent in the ranking swap places even when the full benchmark is run.
The finding that fewer examples suffice does not license dropping datasets or subscenarios; the choice of scenarios is the least reliable decision studied, so trimming tasks is the wrong way to save compute.
DIoR measures reliability, not validity: a benchmark can obtain a high DIoR score while still failing to answer the question it claims to answer.
The claim that Mean Win Rate is gameable is about the metric's structure, not an accusation that any HELM submission was manipulated.
Flash-HELM's up-to-200x saving is a per-tier figure: models expected to land near the top of the ranking still require large sub-samples, and only low-ranked models are evaluated at the cheapest setting.

Terminology in this paper

DIoR: Decision Impact on Reliability: the lower bound of a 95% confidence interval on a similarity meta-metric between benchmark outcomes computed under randomly resampled instantiations of a design decision, such as which datasets or examples are used.
Reliability (of a benchmark): The degree to which a benchmark's answer stays consistent under different equally valid random decisions about its composition, as distinct from validity, which is whether the benchmark answers the intended question.
Scenario / subscenario (HELM): In HELM, a subscenario is an individual dataset with its own scoring function and few-shot prompts, and a scenario is a group of subscenarios weighted together as one unit in the bottom-line score.
Mean Win Rate (MWR): A Borda-count-style benchmark score that averages, over scenarios, the fraction of other evaluated models a given model beats, rather than reporting an absolute task score.
Rank resolution: The number of ranking positions within which a model's benchmark rank can be trusted, used to set how large an evaluation sub-sample a model in a given rank tier requires.
Objective (of a benchmark): The specific question a benchmark is meant to answer, such as obtaining a full model ranking, identifying the best model, or measuring absolute model quality, each of which has its own reliability requirements.

How to cite

@inproceedings{DBLP:conf/naacl/PerlitzBGAESSSC24,author       = {Yotam Perlitz and
                  Elron Bandel and
                  Ariel Gera and
                  Ofir Arviv and
                  Liat Ein{-}Dor and
                  Eyal Shnarch and
                  Noam Slonim and
                  Michal Shmueli{-}Scheuer and
                  Leshem Choshen},
  editor       = {Kevin Duh and
                  Helena G{\'{o}}mez{-}Adorno and
                  Steven Bethard},
  title        = {Efficient Benchmarking (of Language Models)},
  booktitle    = {Proceedings of the 2024 Conference of the North American Chapter of
                  the Association for Computational Linguistics: Human Language Technologies
                  (Volume 1: Long Papers), {NAACL} 2024, Mexico City, Mexico, June 16-21,
                  2024},
  pages        = {2519--2536},
  publisher    = {Association for Computational Linguistics},
  year         = {2024},
  url          = {https://doi.org/10.18653/v1/2024.naacl-long.139},
  doi          = {10.18653/V1/2024.NAACL-LONG.139},
  timestamp    = {Thu, 29 Aug 2024 17:13:57 +0200},
  biburl       = {https://dblp.org/rec/conf/naacl/PerlitzBGAESSSC24.bib},
  bibsource    = {dblp computer science bibliography, https://dblp.org}
}

References

See the full reference list in the paper.